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Preface 


This  monograph  documents  RAND  Corporation  research  on  fighter 
pilot  absorption ,  the  process  of  ensuring  that  a  pilot  new  to  a  weapon 
system  gains  sufficient  experience  with  that  system  to  carry  out  combat 
missions.  After  describing  the  role  of  the  operational  training  environ¬ 
ment  in  producing  experienced  pilots  and  the  benefits  of  training  with 
high-fidelity  simulators  in  mission  training  centers  (MTCs),  we  focus 
on  survey  results  that  show  what  factors,  besides  flying  hours,  contrib¬ 
ute  to  the  development  of  pilot  experience.  The  surveys  on  which  much 
of  this  document  is  based  were  conducted  between  October  2002  and 
October  2003. 

The  research  reported  here  was  sponsored  by  the  Deputy  Chief  of 
Staff  for  Air  and  Space  Operations,  Headquarters  U.S.  Air  Force  (AF/ 
A3/5);  his  counterparts  in  Air  Combat  Command  and  Air  Mobility 
Command;  and  the  Deputy  Chief  of  Staff  for  Manpower  and  Person¬ 
nel,  Headquarters  U.S.  Air  Force  (AF/A1).  It  is  part  of  a  larger  fiscal 
year  2006  study,  “Aircrew  Management  Issues,”  conducted  within  the 
Manpower,  Personnel  and  Training  Program  of  RAND  Project  AIR 
FORCE. 

This  monograph  should  be  of  interest  to  those  involved  in  the 
development  of  training  and  career  management  policies  for  the  Air 
Force. 
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Summary 


One  of  the  main  responsibilities  of  an  operational  fighter  unit  is  to  turn 
inexperienced  pilots  entering  the  unit  into  experienced  pilots  who  are 
able  to  carry  out  the  unit’s  operational  mission  effectively.  The  process 
of  turning  inexperienced  pilots  into  experienced  pilots  is  called  absorp¬ 
tion.  The  Air  Force  must  manage  pilot  absorption  to  achieve  two  goals. 
First,  it  must  ensure  that  operational  units  have  enough  experienced 
pilots  to  perform  the  unit’s  mission  and  to  sustain  the  development 
of  pilots  for  supervisory  flying  positions  in  the  unit.  Second,  it  must 
ensure  that  pilots  gain  the  experience  they  will  need  to  perform  duties 
in  nonflying  positions  that  require  rated  officers. 

Managing  the  absorption  and  subsequent  development  process 
effectively  requires  measuring  pilot  experience.  Since  pilot  experience 
is  developed  through  training  in  the  unit’s  aircraft,  it  is  currently  mea¬ 
sured  in  terms  of  aircraft  flying  hours;  pilots  with  more  flying  hours  are 
presumed  to  be  more  experienced.  The  Air  Force  uses  the  flying-hour 
measure  of  experience  in  its  Rated  Distribution  and  Training  Manage¬ 
ment  (RDTM)  system  as  a  basis  for  maintaining  an  appropriate  mix  of 
experienced  and  inexperienced  pilots  in  operational  units.  The  funda¬ 
mental  RDTM  criterion  for  being  identified  as  “experienced”  is  that  a 
fighter  pilot  have  500  hours  in  the  primary  mission  aircraft.  Although 
the  500-hour  RDTM  definition  is  clear  and  quantitative,  it  may  not 
accurately  reveal  actual  pilot  experience  in  terms  of  skills  or  qualifica¬ 
tion  levels.  The  evidence  is  compelling  that  the  quality  of  500-hour 
pilots  has  varied  considerably  over  the  more  than  30  years  that  the 
definition  has  been  in  use.  These  variations  in  pilot  skills  have  changed 
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the  quality  of  the  operational  training  environment.  Still,  the  RDTM 
definition  has  served  the  Air  Force  well  despite  these  variations  because 
it  has  historically  provided  aircrew  managers  with  a  useful  measure  of 
unit  health  in  terms  of  pilot  qualifications — the  unit’s  experience  level 
(the  ratio  of  experienced  primary  mission  pilots  assigned  divided  by 
total  primary  pilots  assigned).  Embedded  in  the  RDTM  definition  is 
the  assumption  that,  if  the  unit’s  experience  level  remains  high  enough, 
pilots  can  continue  to  develop  the  skills  and  qualifications  required 
to  perform  duties  as  instructors,  supervisors,  and  staff  officers.  This 
assumption  was  quite  valid  when  the  RDTM  definition  was  adopted, 
but  evidence  is  growing  that  the  absorption  and  development  processes 
may  have  become  decoupled  and  may  require  independent  monitoring 
by  aircrew  and  assignment-process  managers.  The  problems  result  from 
inefficiencies  caused  by  increasing  numbers  of  inexperienced  pilots  and 
decreasing  numbers  of  operational  units. 

One  indication  of  possible  inefficiencies  in  the  absorption  process 
is  the  stress  on  operational  units  lacking  sufficient  training  resources  to 
produce  enough  experienced  pilots  (see  Taylor,  Moore,  and  Roll,  2000; 
Taylor  et  ah,  2002;  and  Bigelow  et  ah,  2003).  This  stress  is  imposed, 
to  some  extent,  by  the  RDTM  definition  of  an  experienced  pilot  as 
one  with  at  least  500  hours  in  the  primary  mission  aircraft.  A  more- 
accurate  accounting  of  pilot  absorption  and  development  opportuni¬ 
ties  is  required. 

The  present  research  was  designed  to  examine  how  well  the 
RDTM  flying-hour  measure  of  experience  corresponds  to  expert  pilots’ 
notions  of  what  constitutes  an  experienced  pilot  in  terms  of  ability  to 
perform  additional  tasks,  such  as  staff  or  supervisory  duties.  To  study 
what  is  meant  by  experience ,  it  is  necessary  to  be  able  to  measure  pilots’ 
actual  capability  levels,  which  cannot  be  observed  directly.  Ultimately, 
the  degree  to  which  a  pilot  reflects  an  experienced  ability  level  can  be 
known  only  through  subjective  judgments  made  by  peers  and  com¬ 
manding  officers.  To  determine  the  basis  of  these  judgments,  we  devel¬ 
oped  a  survey  to  determine  the  factors  that  contribute  to  judgments 
of  pilot  experience  and  development.  These  surveys  were  conducted 
between  October  2002  and  October  2003.  Fighter  training  experts 
(supervisors  and  instructor  pilots)  judged  the  experience  level  of  pilots 
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with  different  backgrounds  and  training  experiences  in  terms  of  air¬ 
craft  proficiency  and  readiness  for  a  staff  assignment. 

The  results  of  an  initial  survey  conducted  at  an  F-16  base  indi¬ 
cated  that  flying  hours  are  only  one  of  the  factors  that  contribute  to 
judgments  of  a  pilot’s  experience  (see  p.  62).  Other  factors  include  a 
pilot’s  previous  flying  experience,  upgrade  level,  and  types  of  training 
sorties  flown  (see  pp.  63-64).  These  results  suggest  that  the  current 
approach  to  measuring  experience  in  terms  of  flying  hours  alone  may 
not  accurately  estimate  a  pilot’s  actual  level  of  experience  because  it 
ignores  many  of  the  external  factors  that  influence  the  development  of 
pilot  experience. 

The  results  of  a  second  survey  conducted  at  F-15C  bases  having 
advanced  simulator  systems  with  MTCs  capable  of  linked  distributed 
mission  operations  indicated  that  certain  types  of  MTC  training,  such 
as  large-force  employment  exercises  (which  involve  interaction  among 
a  number  of  aircrews  in  a  simulated  combat  environment),  contrib¬ 
ute  significantly  to  pilot  experience  in  terms  of  combat  capability  (see 
p.  49).  The  results  of  the  F-15C  survey  also  indicated  that  experience 
is  not  a  yes-or-no  variable  and  must  be  considered  in  context.  Having 
the  experience  necessary  not  to  require  special  in-flight  supervision, 
for  example,  is  quite  distinct  from  having  the  experience  required  to 
assume  staff  or  supervisory  duties  (see  p.  42).  Our  survey  results  indi¬ 
cate  that  unit  supervisors  believe  the  latter  type  of  experience  requires 
nearly  twice  the  flying  hours  as  the  500-hour  RDTM  requirement  for 
experience  (see  p.  44).  These  results  strongly  indicate  that  the  absorp¬ 
tion  and  developmental  processes  for  pilots  need  to  be  monitored  inde¬ 
pendently  for  the  Air  Force  to  ensure  its  pilots  have  the  opportunity  to 
continue  to  develop  required  operational  skills  after  their  initial  opera¬ 
tional  assignment. 

The  F-15C  survey  also  asked  about  the  optimal  mix  of  sortie  types 
required  to  maintain  aircraft  proficiency  in  terms  of  combat  readiness. 
The  experts  in  the  F-15C  units  agreed  that  the  optimal  number  of  live 
sorties  per  month  was  about  14  (see  p.  50).  These  experts  also  agreed  that 
the  optimal  number  of  MTC  sorties  per  month  was  about  five.  So  the 
experts  agreed  that  a  total  of  about  19  training  profiles  (including  live 
sorties  and  simulator  missions)  per  month  was  optimal  and  that  about 
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26  percent  of  these  sorties  should  be  large  force  employment  exercises 
flown  in  the  MTC  (see  p.  53).  Clearly,  these  advanced  simulator  systems, 
as  they  become  available,  can  be  regarded  as  essential  components  of  the 
operational  training  environment  that  can  also  contribute  significantly  to 
the  initial  absorption  and  ongoing  development  of  experienced  pilots. 

The  survey  results  suggest  that  it  is  time  for  the  Air  Force  to 
consider  revising  its  yes-or-no  view  of  pilot  experience  to  manage  the 
absorption  and  development  processes  separately  for  both  operational 
training  and  career  progression  (see  p.  57).  An  additional  measure  of 
pilot  experience  will  be  necessary  to  reflect  the  skills  and  qualifications 
acquired  to  prepare  for  more-senior  supervisory  or  staff  assignments. 
The  absorption  process  must  still  be  based  on  some  measure  of  experi¬ 
ence,  but  managing  development  and  subsequent  career  progression 
will  require  an  additional  measure  that  accounts  for  development  of 
these  additional  skills  (see  pp.  57-58). 1 

With  certain  caveats,  we  also  recommend  modifying  all  experi¬ 
ence  definitions  to  include  credit  for  appropriate  training  in  advanced 
simulator  systems  with  MTCs  (see  p.  57).  This  change  would  incorpo¬ 
rate  a  very  effective  training  opportunity  in  units  in  which  it  is  avail¬ 
able  and  could  codify  the  training  requirements  and  accomplishments 
necessary  for  a  more-efficient  mix  of  live  and  simulator  training.  Our 
caveats  include  the  following: 

1 .  Advanced  systems  should  be  available  to  all  pilots  working  with 
given  a  mission  design  series. 

2.  Funding  for  fielding  new  systems,  providing  ongoing  essential 
software  updates  for  fielded  systems,  and  continuing  develop¬ 
ment  of  distributed  links  to  other  appropriate  units  needs  to  be 
adequate. 

3.  Training  requirements  need  to  emphasize  and  exploit  an  inte¬ 
grated  and  coordinated  mix  of  live  and  simulator  training. 


1  The  Air  Force  has  recently  established  officer  development  programs  in  all  career  fields 
to  ensure  senior-level  review  of  officer  development  options  and  opportunities.  We  are  con¬ 
cerned,  however,  that  this  review  process  may  not  begin  early  enough  for  rated  officers  to 
ensure  that  all  (or  even  most)  of  them  can  continue  to  develop  the  required  operational  skills 
in  subsequent  assignments  after  their  initial  operational  assignments. 
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aircraft 
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ACT 

air  combat  tactics 
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air  expeditionary  force 

AFB 
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AFRF 

Air  Force  Research  Faboratory 
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basic  flight  maneuvers 
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distributed  mission  operations 

DMT 

distributed  mission  training 

FY 
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MDS 

mission  design  series 

FFE 

large  force  employment 
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MTC 

mission  training  center 

PAA 

primary  aircraft  authorized 

PMA 

primary  mission  aircraft 

PMAI 

primary  mission  aircraft  inventory 

RDTM 

Rated  Distribution  and  Training  Management  System 

SCM 

sorties  per  crew  per  month 

UTE 

aircraft  utilization 

CHAPTER  ONE 


Introduction 


Operational  fighter  units  must  provide  the  training  necessary  to  turn 
inexperienced  pilots  beginning  their  initial  operational  assignment  in 
a  new  weapon  system  into  experienced  pilots  who  can  perform  the 
unit’s  specific  combat  mission.  This  training  must  also  prepare  pilots  to 
continue  to  acquire  the  skills  required  to  fill  rated  supervisory  and  staff 
positions  at  the  wing  level  and  above.1  The  process  of  turning  pilots 
new  to  a  weapon  system  into  experienced  pilots  is  called  absorption. 

The  Air  Force  must  manage  pilot  absorption  to  achieve  two  goals. 
First,  it  must  ensure  that  pilots  new  to  a  unit  are  able  to  meet  the  expe¬ 
rience  criterion  within  the  time  available  in  their  initial  operational 
flying  opportunity.  Second,  it  must  ensure  that  units  have  enough 
qualified  pilots  to  perform  the  unit’s  mission  and  to  provide  adequate 
in-flight  supervision  and  instruction  to  the  pilots  in  the  unit  with  less 
experience  and  fewer  qualifications.  The  latter  requirement  relates 
directly  to  helping  officers  acquire  the  additional  operational  skills  and 
qualifications  they  need  to  fill  the  supervisory  and  staff  positions  that 
the  Air  Force  has  identified  as  requiring  rated  officers. 

For  several  decades,  the  Air  Force  has  used  a  pilot’s  flying  hours 
as  a  measure  of  experience  to  manage  both  tasks  through  the  Rated 
Distribution  and  Training  Management  (RDTM)  system,  which,  as 
described  in  more  detail  in  Chapter  Two,  defines  an  experienced  pilot 
as  one  having  500  hours  in  the  unit’s  primary  mission  aircraft.  But  the 


1  The  unit  conducts  training  to  help  maintain  currency  in  perishable  skills,  to  provide 
opportunities  to  increase  proficiency  and  operate  in  a  wider  range  of  mission  contexts,  and 
to  help  sustain  unit  combat  capability. 
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developmental  process  clearly  includes  exposure  to  contexts  beyond, 
and  substantially  different  from,  the  experiences  and  situations  encoun¬ 
tered  in  a  pilot’s  initial  500  hours  of  flying. 

Chapter  Two  describes  the  operational  training  environment  and 
changes  in  the  quality  and  quantity  of  flying  training  since  the  1970s 
that  may  undermine  some  of  the  original  assumptions  that  made  the 
RDTM  system  valid.  In  particular,  the  chapter  discusses  the  impor¬ 
tance  of  monitoring  the  relationship  between  the  problems  of  absorb¬ 
ing  individual  pilots  and  of  the  unit’s  training  and  development. 

Chapter  Three  describes  the  MTC  training  facilities  and  high- 
fidelity  simulator  systems  that  are  now  available  in  some  fighter  units 
and  outlines  how  they  can  be  used  efficiently  to  provide  certain  types 
of  training.  Chapter  Four  describes  the  surveys  we  conducted  between 
October  2002  and  October  2003  to  understand  training  experts’  views 
of  what  constitutes  experience  in  different  contexts  for  pilots.  Chapter 
Five  discusses  the  implications  of  the  survey  results  and  addresses  the 
aircrew  management  policy  changes  that  these  results  suggest  will  be 
required. 


CHAPTER  TWO 


The  Role  of  the  Operational  Training 
Environment 


Operational  units  provide  the  resources  required  for  each  Air  Force 
weapon  system  to  accomplish  its  primary  mission.  For  fighter  units, 
this  means  that  aircrews  assigned  to  these  units  must  continually  train 
to  develop  and  maintain  their  combat  capabilities.  But  operational 
units  also  have  another  important  function:  They  must  continue  to 
sustain  themselves  by  turning  new  pilots  into  experienced  pilots  and 
ensuring  that  they  have  the  opportunity  to  continue  developing  into 
flight  leads,  instructor  pilots,  and  supervisors. 


The  Training  Environment 

Earlier  RAND  Corporation  work  has  addressed  the  degrading  effects 
that  low  experience  (inadequately  qualified  pilots)  and  overmanning 
have  in  all  types  of  operational  fighter  units  (see  Taylor  et  ah,  2002, 
and  Taylor,  Moore,  and  Roll,  2000).  Other  factors  can  also  affect  unit 
training  effectiveness  and  therefore  the  development  of  new  pilots.  In 
the  next  few  paragraphs,  we  will  examine  the  operational  training  envi¬ 
ronment  more  carefully  to  better  understand  the  causes  (and  effects) 
of  some  of  these  factors.  Our  purpose  is  to  provide  a  useful  framework 
that  will  enable  us  to  examine  more  comprehensively  the  contributions 
that  high-fidelity  simulators  and  MTCs  can  make  to  the  overall  opera¬ 
tional  environment.  For  specificity,  we  will  focus  primarily  on  F-15C 
air-superiority  units  because  they  were  the  first  Air  Force  fighter  units 
to  receive  the  advanced  simulators  and  MTCs. 


3 


4  Absorbing  and  Developing  Qualified  Fighter  Pilots 


Training  Basics 

The  fundamental  training  unit  for  fighters  is  a  single  sortie;  each  sortie 
is  typically  devoted  to  a  specific  mission  need.  These  mission  training 
sorties  often  build  on  one  another  inasmuch  as  the  skills  stressed  in 
one  type  of  training  sortie  will  be  essential  components  of  more-com¬ 
plex  training  sorties.  Figure  2.1  illustrates  this  building-block  approach 
by  using  an  F-15C  air-superiority  unit’s  primary  training  sorties  as  an 
example.  Advanced  handling  characteristics  sorties  are  the  most  funda¬ 
mental  of  the  building  blocks.  In  such  a  sortie,  the  pilot  flies  a  single 
aircraft,  with  no  adversary,  to  fully  explore  the  aircraft’s  maximum- 
performance  maneuvering  envelope.  This  develops  familiarity  with  the 
flight  maneuvers  that  will  be  required  in  an  air-to-air  combat  envi¬ 
ronment.  It  also  helps  pilots  recognize  key  in-flight  indications  that 
enable  maneuvering  at  maximum  efficiency  and  avoid  departing  from 
controlled  flight. 

The  remaining  training  sorties  each  introduce  in  turn  new  skills 
that  will  be  essential  in  an  air-to-air  combat  engagement.  Basic  fighter 
maneuver  (BFM)  sorties,  for  example,  stress  the  techniques  needed  for 
a  single  aircraft  to  maneuver  successfully  against  a  single  adversary  in 
a  variety  of  settings,  while  air  combat  maneuver  (ACM)  sorties  stress 

Figure  2.1 

Building-Block  Model  of  Training  Missions  for  the  F-15C 
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maneuvering  a  two-ship  element  against  an  adversary  to  maintain 
mutual  support  and  fully  appreciate  a  larger  portion  of  the  threat  spec¬ 
trum.  Tactical  intercepts  enable  four-ship  flight  members  to  master  the 
essential  cues,  indications,  and  geometric  relations  for  using  radar  and 
other  nonvisual  methods  to  sort,  target,  and  engage  a  flight  of  adver¬ 
sary  aircraft.  Air  combat  tactics  (ACT)  sorties  enable  four-ship  flights 
to  explore  a  number  of  realistic  combat  scenarios  against  two  or  more 
adversaries.  Finally,  large  force  employment  (LFE)  exercises  enable  mul¬ 
tiple  four-ship  flights  to  understand  the  specific  roles  and  flight  interre¬ 
lationships  involved  in  meeting  all  the  responsibilities  required  for  dif¬ 
ferent  kinds  of  specific  missions.  These  preplanned  exercises  simulate 
realistic  combat  employment  requirements. 


Demands  That  Can  Degrade  Training  Quality 

An  F-15C  unit  must  meet  other  sortie  demands  that  are  not  listed 
among  the  primary  training  sorties  in  Figure  2.1.  Some  of  these  provide 
essential  training,  such  as  instrument  and  ocean-crossing  deployment 
sorties,  and  can  be  quite  useful  unless  they  are  being  performed  merely 
because  weather  or  other  constraints  have  forced  them  to  replace  pri¬ 
mary  training  sorties.  Other  sortie  demands  that  one  might  expect  to 
be  useful  can  actually  degrade  training. 

Contingency  Operations 

Flying  requirements  that  support  actual  contingency  operations,  such 
as  Southern  Watch  (flown  in  Iraq  between  the  Gulf  War  and  Iraqi 
Freedom),  Noble  Eagle  (flown  as  post-9/11  homeland  security  support 
in  the  United  States),  and  Enduring  Freedom  (flown  in  Afghanistan), 
can  provide  useful  training  initially,  but  participating  pilots  generally 
agree  that  the  training  can  become  quite  diluted  when  the  same  mis¬ 
sions  are  flown  repetitively.  A  primary  factor  for  air-superiority  units, 
of  course,  has  been  the  total  absence  of  enemy  fighters  during  these 
operations.  Another  factor  is  that  the  participating  aircraft  must  be 
loaded  with  live  ordnance,  thereby  significantly  limiting  the  available 
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training  options  for  these  sorties  and  virtually  eliminating  the  oppor¬ 
tunity  to  use  these  sorties  to  conduct  pilot  upgrade  flights. 

It  is  important  to  note  that  flying  hours  devoted  to  contingency 
operations  reduce  the  total  flying  hours  available  for  a  unit’s  primary 
training  needs.  Congressional  rules  require  active  fighter  units  to  “fly 
out”  all  their  allocated  training  hours  for  a  given  fiscal  year  (FY)  before 
the  Air  Force  can  request  supplemental  funding  for  contingency  hours. 
This  often  means  that  the  supplemental  hours  cannot  be  requested  and 
approved  in  time  to  be  flown  during  the  required  fiscal  year.1 

Requirements  for  Adversary  Aircraft 

Another  factor  that  takes  sorties  away  from  the  fundamental  train¬ 
ing  regimen  and  inhibits  training  opportunities  in  operational  units 
is  the  requirement  for  units  to  provide  their  own  adversary  aircraft  for 
most  of  their  air-superiority  training  sorties.  These  are  called  red  air 
sorties ,  and  the  participants  are  expected  to  emulate  recognized  enemy 
tactics  and  weapon  system  capabilities  as  faithfully  as  possible.  This 
often  means  the  red  air  pilots  cannot  fully  exploit  the  advanced  tac¬ 
tics,  maneuvering  envelope,  and  state-of-the-art  cueing  systems  that 
the  blue  air  sorties  in  the  normal  training  regimen  are  designed  to  build 
on.  Thus,  while  pilots  flying  red  air  sorties  do  increase  their  total  flying 
hours,  they  do  not  improve  their  skills  as  much  as  if  they  were  flying 
blue  air  sorties. 

Indeed,  according  to  instructor  pilots  and  supervisors,  these  sor¬ 
ties  can  generate  negative  learning  circumstances  for  pilots  in  their  ini¬ 
tial  stages  of  development.  For  example,  after  completing  initial  mis¬ 
sion  qualification  training,  many  new  pilots  in  units  whose  training 
sorties  are  at  a  premium  discover  that  their  “wingman”  status  means 
a  scheduling  priority  so  low  that  the  primary  way  to  get  on  the  flying 
schedule  at  all  is  to  fly  red  air  sorties.  This  is  likely  to  continue  until  the 
pilots  develop  their  skills  enough  to  upgrade  to  a  higher  qualification 
level.  This  can  take  an  extended  time  under  any  circumstances  when 


1  This  information  about  supplemental  funding  rules  was  provided  by  AF/XOOT  (now 
AF/A30T)  and  ACC/DOT  (now  ACC/A3T)  staff  members.  We  will  forego  further  discus¬ 
sion  of  these  complicated  rules  to  avoid  straying  too  far  from  our  intended  topic. 
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sorties  are  scarce,  but  the  problem  is  clearly  exacerbated  because  red  air 
sorties  do  not  contribute  as  effectively  to  pilot  development  as  would 
the  same  number  of  flying  hours  of  blue  air  sorties. 

The  requirement  to  devote  flying  hours  to  sorties  that  provide 
less- effective  training  provides  a  simple  example  of  the  factors  that  can 
degrade  the  operational  training  environment.  Training  effectiveness 
in  operational  fighters  has  indeed  varied  considerably  over  time.  A  look 
at  some  of  the  other  factors  that  generated  previous  fluctuations  will  be 
useful  for  examining  the  options  that  may,  or  may  not,  be  effective  in 
dealing  with  the  current  issues. 


Historical  Fluctuations  in  the  Effectiveness  of  the  Training 
Environment 

We  begin  this  review  with  the  cessation  of  hostilities  in  Southeast  Asia. 
The  resulting  reduction  in  operational  fighter  units,  coupled  with  the  lead 
time  required  to  reduce  the  production  of  new  aircrew  members  exit¬ 
ing  from  an  expanded  wartime  training  pipeline,  had  left  the  remaining 
units  with  excessive  numbers  of  very  recently  trained  (and  inexperienced) 
pilots.  Modern  aircrew  management  methods  began  with  the  RDTM 
system,  which  was  introduced  to  address  these  problems. 

The  Role  of  RDTM 

The  RDTM  system  addressed  the  experience  and  manning  problems  by 
first  identifying  the  absorption  problem  as  the  ability  for  units  within  a 
weapon  system  “to  accept  new  pilots  and  maintain  an  acceptable  expe¬ 
rience  level.”  It  provided  more-meaningful  definitions  for  experienced 
pilots,  established  acceptable  experience-level  criteria  for  operational 
units,  and  provided  quantitative  methods  for  managing  training  pipe¬ 
lines  and  assignment  processes  to  ensure  that  future  inventories  could 
match  projected  requirements.2 


2  The  Air  Force  implemented  the  RDTM  system  via  Program  Guidance  PG-77-1  (1975); 
see  especially  Section  C,  paragraph  4-10,  pp.  4-20).  Taylor  et  al.  (2002,  p.  40)  discusses  the 
system’s  origin  in  more  detail. 
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The  fundamental  RDTM  criterion  that  identifies  a  fighter  pilot  as 
“experienced”  is  500  hours  in  the  primary  mission  aircraft  (PMA),  and 
the  system’s  implementing  document  defines  unit  experience  levels  as 
the  unit’s  number  of  experienced  primary  mission  pilots  (aircrew  posi¬ 
tion  indicator  1  [API-1])  divided  by  the  total  number  of  API-1  pilots 
assigned  to  the  unit.  The  system  then  used  unit  experience  levels  as  the 
primary  measure  of  the  unit’s  ability  to  continue  to  sustain  itself  by 
developing  qualified  flight  leads,  instructor  pilots,  and  supervisors  to 
replace  qualified  individuals  who  move  on  to  important  staff,  manage¬ 
ment,  and  command  billets  commensurate  with  increasing  rank  and 
qualifications.3  The  developmental  process  clearly  includes  the  expe¬ 
rience  pilots  gain  in  contexts  that  may  differ  substantially  from  that 
obtained  solely  in  the  pilot’s  initial  500  hours  of  flying. 

The  RDTM  implementation  also  improved  the  aircrew  manage¬ 
ment  process  because  the  steady-state  sustainment  rates  (the  annual 
new  pilot  production  rates  required  to  sustain  future  requirements 
and  maintain  unit  experience  levels)  could  now  be  calculated  by  major 
weapon  system  and  actual  mission  design  series  (MDS).  This,  in  turn, 
enabled  the  assignment  process  to  control  unit  experience  levels  by 
matching  the  flow  of  new  pilots  who  were  completing  their  initial 
training  with  adequate  numbers  of  experienced  pilots  to  ensure  that 
prespecified  experience-level  criteria  were  met.  This  procedure  quan¬ 
tified  the  absorption  problem  and  provided  a  management  approach 
with  which  to  address  it.  In  this  context,  the  notion  of  unit  experience 
was  identified  with  unit  health  in  interpreting  whether  the  unit  could 
maintain  adequate  numbers  of  assigned  pilots  who  were  qualified  as 
flight  leads,  instructor  pilots,  and  supervisors  to  conduct  its  required 
training  and  simultaneously  distribute  sorties  relatively  equitably  to  its 
assigned  pilots.4 


3  A  pilot  who  has  previously  qualified  in  another  Air  Force  aircraft  is  considered  experi¬ 
enced  in  his  or  her  primary  fighter  aircraft  after  flying  it  for  300  hours  if  their  total  flying 
time  is  at  least  1,000  hours.  This  most  commonly  occurs  with  first-assignment  instructor 
pilots,  who  initially  instruct  in  undergraduate  pilot-training  programs.  Later  references  to 
the  500-hour  criterion  for  experienced  pilots  assume  this  proviso. 

Taylor  et  al.  (2002)  provides  a  more-complete  discussion  of  absorption  issues. 
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Aircraft  Utilization  and  Training  Quantity 

Training  effectiveness  is  clearly  a  function  of  both  the  quality  and 
quantity  of  the  available  training.  With  higher  manning  levels,  for 
example,  if  the  unit’s  total  number  of  sorties  available  remains  relatively 
constant,  the  assigned  pilots  will  average  fewer  sorties  per  crew  per 
month  (SCM).5  Low  unit  experience  levels  mean  that  fewer  pilots  have 
instructor  pilot  qualifications,  for  example,  and  they  will  individually 
need  to  fly  more  sorties  per  month  to  provide  in-flight  supervision  than 
are  required  to  maintain  their  own  proficiency.  The  additional  sorties 
stem  from  unit  training  needs  driven  by  upgrades  and  other  supervi¬ 
sory  responsibilities.  When  highly  qualified  pilots  must  be  tasked  to 
fly  more  sorties  than  they  require  individually,  less-qualified  pilots  will 
clearly  share  fewer  sorties.  As  a  result,  these  pilots  will  have  lower  SCM 
averages  when  sortie  resources  cannot  be  increased.6 

The  sortie  pool  available  to  fighter  units  depends  primarily  on  the 
units’  aircraft  utilization  (UTE)  rates.  UTE  rates,  which  are  a  measure 
of  maintenance  effectiveness,  are  defined  as  the  average  number  of  sor¬ 
ties  a  unit  flies  per  airframe  per  month.  Here,  the  airframes  counted 
correspond  to  the  unit’s  primary  mission  aircraft  inventory  (PMAI).7 
Following  the  post-Southeast  Asia  drawdown,  the  average  number  of 
training  sorties  available  to  the  new,  and  thus  less  experienced,  pilots 
decreased  because  the  number  of  operational  units  was  decreased 
abruptly.  This  was  the  problem  that  led  to  the  implementation  of  the 
RDTM  system.  Part  of  the  cause  was  the  overmanning  and  the  low 
experience  and  qualification  factors  that  we  have  already  discussed. 
Low  sortie  counts  and  associated  flying-hour  problems  remained,  how¬ 
ever,  even  after  the  RDTM-induced  aircrew  management  improve- 


6  Taylor  et  al.  (2002)  addresses  the  quantitative  effect  of  higher  manning  levels. 

6  As  an  exaggerated  example,  if  20  inexperienced  pilots  in  a  unit  each  need  a  sortie  with 
an  instructor  pilot  to  upgrade  to  two-ship  flight  lead  and  if  there  is  only  one  instructor  pilot 
in  the  unit,  the  instructor  pilot  will  have  to  fly  20  sorties  to  provide  one  sortie  to  each  of  the 
inexperienced  pilots. 

7  Each  active  fighter  squadron  currently  has  either  18  or  24  primary  mission  aircraft  in  its 
inventory  (i.e.,  18-PMAI  or  24-PMAI). 
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ments  had  eliminated  virtually  all  the  manning  and  experience  issues 
in  the  units.8 

Indeed,  even  the  operational  implementation  of  a  sizable  aircraft 
modernization  program  in  the  1970s — which  included  the  addition  of 
the  F-15A/C  to  the  fighter  force,  followed  closely  by  the  A-10,  the  F-16, 
and  the  F-15E — did  not  improve  the  limited  availability  of  training 
sorties  and  flying  hours.  Nevertheless,  the  average  flying  hours  per  pilot 
continued  to  decline  for  several  years.  In  the  face  of  rapidly  increas¬ 
ing  airline  hiring,  experienced  fighter  pilots  voted  with  their  feet  and 
walked  over  to  commercial  aviation  in  significant  numbers.  However, 
the  Air  Force  was  able  to  fly  itself  out  of  the  resulting  decline  in  pilot 
experience,  with  an  impressive  reversal  in  the  flying-hour  trend.  Along 
with  the  new  fighters  came  a  steady  increase  in  UTE  rates  and  flying 
hours.  The  resulting  increase  in  sortie  resources,  coupled  with  the 
improved  aircrew  management  options  the  RDTM  system  brought, 
enabled  an  increase  in  the  average  fighter  pilot’s  annual  flying  hours 
from  150  to  230  between  the  mid-1970s  and  the  mid-1980s.  Fur¬ 
thermore,  the  more  than  50-percent  increase  in  flying  hours  mainly 
involved  the  more-complex  areas  of  the  training  spectrum.9 

We  have  seen  that  factors  such  as  low  UTE  rates  and  sorties  that 
do  not  contribute  to  the  development  or  maintenance  of  flying  skills 
can  degrade  the  training  environment,  and  we  have  seen  that  improved 
aircrew  management  methods  increased  UTE  rates,  which  helped 
overcome  the  post-Vietnam  training  problems  that  developed  in  opera¬ 
tional  fighter  units. 

Additional  Training  Environment  Improvements 

Several  other  training  innovations  also  contributed  significantly  toward 
improving  the  operational  training  environment.  On  the  eve  of  the 


8  Much  of  the  historical  information  presented  in  this  and  subsequent  paragraphs  is  derived 
from  Anderegg  (2001).  For  additional  historical  context,  see  Lambeth  (2000). 

9  Gen  William  Creech  became  commander  of  Tactical  Air  Command  in  1978  and  started 
a  gradual  recovery  from  what  he  termed  the  “slippery  slope”  of  declining  UTE  rates  and 
flying  hours  per  pilot.  For  a  complete  discussion  of  how  his  management  style  improved  the 
command’s  flying  program,  see  Peters  et  al.  (1989).  The  flying  increase  for  fighter  pilots  is 
documented  in  Lambeth  (2000,  p.  71). 
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1991  Gulf  War,  the  U.S.  Air  Force  fighter  force  was  strong  because 
several  initiatives  had  come  to  fruition  over  the  preceding  15  years. 
Along  with  the  new  jets  and  increased  flying  hours  came  new  training 
initiatives — most  of  which  were  aimed  at  the  high  end  of  the  training 
spectrum.  Although  there  were  many  such  initiatives,  four  stand  out  as 
the  most  important:  the  Red  Flag  initiative,  the  aggressor  program,  the 
addition  of  air  combat  maneuvering  and  instrumentation  (ACMI),  and 
an  emphasis  on  the  use  of  gun  cameras  and  cockpit  video  recorders. 

Red  Flag.  Analysis  of  the  rate  of  combat  losses  compared  to  the 
number  of  airmen  in  all  wars  indicated  a  common  thread:  If  the  new 
pilot  survived  the  first  ten  missions,  he  was  likely  to  survive  his  combat 
tour.  Col  Moody  Suter,  commonly  acknowledged  as  the  father  of  Red 
Flag,  devised  a  large  training  exercise,  complete  with  a  realistic  array 
of  defenses,  using  the  Nellis  Air  Force  Base  (AFB),  Nevada,  training 
ranges.  Beginning  in  1977,  Red  Flag  gave  young  pilots  the  important 
first  ten  missions  in  peacetime.  By  the  start  of  the  Gulf  War,  thousands 
of  pilots  had  been  “blooded”  at  Red  Flag  in  a  safe  and  closely  super¬ 
vised  environment.  The  success  of  Red  Flag  is  attributed  to  four  main 
features: 

1.  a  relatively  unconstrained  volume  of  airspace  in  which  pilots 
could  maneuver  realistically 

2.  a  functioning,  integrated  air  defense  system  based  on  the  Soviet 
model 

3.  the  opportunity  to  train  in  large  force  packages  similar  to  those 
planned  for  combat 

4.  a  comprehensive  debriefing  system  that  could  accurately  mea¬ 
sure  everything  from  an  individual  pilot’s  performance  to  an 
assessment  of  the  overall  tactical  plan. 

Aggressors.  The  aggressor  program  consisted  of  pilots  and  fighter 
aircraft  specifically  trained  and  equipped  to  replicate  the  Soviets  and 
the  Soviet  system  and  to  oppose  the  pilots  in  operational  units.  It  was 
the  beginning  of  a  culture  of  learning  to  fight  dissimilar  aircraft  that 
use  dissimilar  tactics.  The  aggressors  not  only  provided  adversary  sup- 
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port  for  Red  Flag  but  also  conducted  regular  and  frequent  “road  shows” 
at  fighter  bases  to  train  pilots  in  smaller  scenarios. 

Air  Combat  Maneuvering  and  Instrumentation.  The  ACMI 
system  employs  a  missile-sized  pod  on  the  fighter  and  a  linked  group 
of  receiving  stations  and  computers  on  the  ground  that  capture  every 
move  a  fighter  makes  in  a  dogfight.  The  data  can  then  be  replayed  after 
the  fight  as  a  visual  presentation  that  shows  all  the  fighters  who  shot 
whom,  when,  where,  and  how.  These  replays  accelerated  the  learning 
process  dramatically,  and  soon  the  system  spread  from  Nellis  to  several 
other  sites  around  the  world.  Before  ACMI,  the  victor  in  an  air-to-air 
fight  was  often  the  pilot  who  could  talk  the  fastest  or  wave  his  hands 
the  most,  and  kills  were  claimed  based  on  each  pilot’s  own  interpreta¬ 
tion  of  how  well  employment  parameters  were  met.  After  an  ACMI- 
telemetered  engagement,  however,  the  results  were  irrefutable.  Pilots 
could  see  clearly  during  the  debrief  what  worked  and  what  did  not. 

Gun  Cameras  and  Cockpit  Video  Recorder.  Several  models  of 
the  primary  fighter-bomber  of  the  Vietnam  War,  the  F-4,  had  no  gun 
camera.  Part  of  the  training  culture  that  evolved  along  with  Red  Flag, 
aggressors,  and  ACMI  was  the  return  of  the  gun  camera,  followed  rap¬ 
idly  by  several  iterations  of  video  recorders  to  capture  the  parameters 
of  a  bombing  pass  or  a  simulated  air-to-air  missile  firing.  Similar  to 
ACMI  results,  the  gun  camera  showed  the  pilot  his  precise  position 
and  parameters  at  simulated  or  actual  weapon  release. 

Summary.  Red  Flag,  the  aggressor  program,  ACMI,  gun  cameras, 
and  video  recorders  all  combined  with  the  increasing  availability  of 
flying  hours  for  training  to  improve  both  the  quality  and  quantity  of 
training  and  dramatically  improved  the  operational  training  environ¬ 
ment  that  was  available  at  the  start  of  the  Gulf  War.  The  superior  per¬ 
formance  of  the  Air  Force’s  fighter  force  in  that  war  speaks  to  the  qual¬ 
ity  of  the  prewar  training  program  for  the  participating  pilots.  Since 
the  Gulf  War,  however,  the  Air  Force  has  had  less  “quantity”  training, 
as  seen  in  the  decline  of  sorties  available  for  training. 

Post-Gulf  War  Training  Environment 

The  data  in  Figure  2.2  show  the  decreases,  since  the  Gulf  War,  in  UTE 
rates,  SCM,  hours  per  crew  per  month,  and  the  portion  of  UTE  alio- 
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Figure  2.2 

Air  Combat  Command  F-15A/C  UTE,  Hours,  Sorties  Comparison 


a  Includes  Keflavik,  Iceland,  and  Operations  Northern  Watch  and  Southern  Watch. 
b  Pre-AEF  and  Keflavik. 

RAND  MG597-2.2 


cated  to  home-station  training  for  Air  Combat  Command’s  (ACC’s) 
F-15Cs.10 

The  UTE  rate  is  further  broken  down  into  deployed  UTE  and 
home  UTE.* 11  It  is  important  to  note  that  the  deployed  UTE  rate 
probably  represents  disproportionately  more  hours  than  does  the 
home  UTE  because  many  contingency  sorties  were  flown  as  combat 
air  patrol  or  package  support  sorties  that  averaged  four  to  six  hours 
each,  while  normal  training  sorties  are  programmed  at  only  1.3  hours 
each.  Figure  2.2  shows  that  in  1990,  the  year  before  the  Gulf  War, 
all  the  sorties,  and  thus  all  the  available  flying  hours,  were  devoted 
to  normal  operational  training.  During  the  period  following  the 


10  Data  provided  by  ACC/DOT,  2003. 

11  The  deployed  UTE  includes  contingency  missions  flown  as  part  of  Operation  Noble 
Eagle,  even  if  they  were  flown  from  the  unit’s  home  station,  but  it  does  not  include  sorties 
flown  during  specified  training  deployments,  such  as  Red  Flag. 
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Gulf  War,  however,  the  overall  drop  in  aircraft  utilization,  cou¬ 
pled  with  the  cost  of  the  sorties  devoted  to  contingency  operations, 
reduced  the  average  number  of  sorties  (per  airframe  per  month)  that 
could  be  devoted  to  operational  training  by  some  30  percent.  We 
may  presume  that  the  monthly  flying  hours  devoted  to  normal  train¬ 
ing  per  inexperienced  pilot  decreased  by  at  least  that  amount.  Sim¬ 
ilar  data  for  the  other  fighter  MDSs  reveal  similar  trends,  although 
the  training  value  of  the  contingency  support  flown  after  the  onset 
of  open  hostilities  in  Afghanistan  and  Iraq  could  perhaps  be  argued 
to  have  been  more  valuable  for  these  MDSs  than  for  the  F-15Cs.12 

Experience  and  Qualifications 

RDTM  included  the  implicit  assumption  that,  if  enough  pilots  could 
be  absorbed,  these  pilots  could  continue  their  operational  development 
apace  and  unimpeded  during  subsequent  operational  tours  of  duty.  The 
terminology  that  identifies  a  unit’s  experience  level  with  the  ade¬ 
quacy  of  its  assigned  qualified  pilots  was  completely  valid  when  it 
was  originally  made  because  the  continued  development  of  fighter 
pilots  was  virtually  assured  by  subsequent  operational  assign¬ 
ments.13  Moreover,  the  identification  of  experience  was  further 
recognized  in  the  Ready  Aircrew  Program,  which  specifies  oper¬ 
ational  training  requirements  in  fighter  units.  Except  for  brief  peri¬ 
ods  in  which  the  assignment  system  was  required  to  react  abruptly 
to  significant  policy  changes,  it  remained  valid  for  over  two  decades. 

However,  recent  RAND  analysis  has  raised  strong  concerns  about 
whether  the  assumption  that  aircrews  can  continue  their  operational 
development  once  they  meet  their  weapon  system’s  experience  defini- 


12  As  discussed  previously,  contingency  sorties  do  provide  limited  amounts  of  training  ini¬ 
tially,  but  the  value  of  the  training  degrades  rapidly  with  continued  repetition.  Participating 
pilots  have  typically  considered  the  training  available  from  contingency  operations  for  the 
A/OA-10  to  be  somewhat  more  valuable  than  that  available  for  the  other  fighters  throughout 
the  period  depicted.  See  Stillion  (1999)  for  a  more-comprehensive  treatment  of  aircrew  train¬ 
ing  issues  during  contingency  operations  in  the  1990s. 

13  This  identification  was  also  fully  incorporated  in  the  aircrew  management  analysis  of 
Taylor  et  al.  (2002),  which  addresses  the  distinctions  between  unit  training  needs  and  indi¬ 
vidual  training  summations. 
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tion  is  still  valid.14  As  an  example,  discussions  with  unit  instructor  pilots 
and  supervisors  have  confirmed  that  there  is  a  vast  difference  between 
pilots  who  have  recently  met  the  experience  criterion  and  pilots  who 
are  ready  to  move  into  staff  or  supervisory  billets.  Pilots  in  the  latter 
group  require  a  great  deal  more  experience  than  the  500-hour  criterion 
represents,  and  indeed  the  binary  (yes-or-no)  definition  of  experience 
will  no  longer  be  useful  if  this  continued  development  of  additional 
qualifications  is  no  longer  assured.  The  opportunity  for  these  pilots  to 
continue  their  development  after  meeting  the  initial  500-hour  defi¬ 
nition  so  that  they  can  achieve  the  additional  qualifications  required 
for  moving  on  to  new  roles  and  assignments  is  the  subject  of  ongoing 
RAND  research.15 


What  Does  This  Mean? 

Both  the  quality  and  quantity  of  the  training  available  in  operational 
units  has  changed  considerably  in  the  three  decades  since  the  500-hour 
criterion  was  established  as  part  of  the  RDTM  system.  The  operational 
training  environment  improved  dramatically  in  the  late  1970s  and 
1980s  as  the  Air  Force  increased  aircraft  utilization  rates,  modernized 
its  fighter  force,  and  implemented  a  number  of  training  innovations 
that  definitely  enhanced  the  opportunities  for  pilots  to  train  in  the 
operational  units.  Following  the  first  Gulf  War,  however,  the  train¬ 
ing  environment  began  to  deteriorate  as  overall  UTE  rates  gradually 
decreased  and  a  significant  portion  of  the  hours  inexperienced  pilots 
flew  had  to  be  devoted  to  low-value  contingency  operations.  The  mag- 


14  Taylor  (2006). 

15  Taylor,  Moore,  and  Roll  (2000)  and  Bigelow  et  al.  (2003)  document  the  operational 
training  needs  for  fighter  units.  As  previously  referenced,  the  absorption  issues  are  docu¬ 
mented  in  Taylor  et  al.  (2002).  The  most  obvious  example  of  an  abrupt  policy  change  that 
generated  a  system  reaction  is  the  fall  of  the  Berlin  Wall  and  the  dramatic  drawdown  in 
personnel  and  force  structure  that  ensued.  The  interview  and  survey  data  will  be  discussed 
in  more  detail  later  in  this  report.  Finally,  a  related  project  examining  issues  associated  with 
developing  rated  officers  to  meet  authorized  billet  requirements  has  also  identified  much 
more-formal  qualifications  for  most  staff  officers  and  supervisor  billet  authorizations  than 
the  500-hour  definition. 
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nitude  of  the  contingency  effort  is  clear  from  the  fact  that,  during  the 
10  years  immediately  following  the  Gulf  War,  the  Air  Force  expended 
nearly  400,000  sorties  simply  to  maintain  no-fly  zones  in  northern 
and  southern  Iraq.  The  training  environment  was  further  degraded  by 
the  fact  that  the  active  fighter  force  structure  was  cut  in  half  during 
almost  the  same  period,  while  the  number  of  fighter  pilot  positions 
decreased  by  only  23  percent,  effectively  increasing  the  ratio  of  pilots 
to  aircraft.16 

The  downward  trend  in  overall  UTE  rates  stabilized  in  2001  as  the 
Air  Force  reinstated  standardized  UTE  rates  for  all  combat  air  forces 
aircraft  and  the  contingency  commitment  has  been  slightly  mitigated 
after  a  full  cycle  of  programmed  contingency  support  under  the  air 
expeditionary  force  (AEF)  structure.  However,  the  fact  remains  that, 
even  if  all  available  UTE  were  expended  on  home  training,  the  number 
of  sorties  (and,  thus,  flying  hours)  flown  per  unit  for  training  would 
still  be  significantly  lower  than  it  was  before  the  1991  Gulf  War. 

The  primary  reductions  in  available  training  are  at  the  higher  and 
more-complex  end  of  the  training  spectrum  depicted  in  Figure  2.1. 
This  is  because  safety  of  flight  and  supervisory  concerns  will  always 
demand  suspending  the  more-complex  training  activities  as  pilots 
begin  to  lose  proficiency  in  the  fundamental  skills  required  to  accom¬ 
plish  the  high-end  training.  This,  coupled  with  the  fact  that  flight  leads 
and  instructor  pilots  must  tailor  training  scenarios  to  accommodate 
the  least  proficient  flight  members,  means  that  the  more-difficult  train¬ 
ing  scenarios  become  much  harder  to  schedule  as  the  available  training 
sorties  decrease.  Thus,  the  overall  training  quality  will  generally  drop 
as  the  quantity  of  training  available  decreases.  The  total  flying  hours 
available  to  a  unit  is  directly  proportional  to  the  unit’s  average  UTE 
rate.  In  today’s  environment,  the  combat  air  forces’s  standard  UTE 
rates  (essentially  the  highest  rates  logistics  community  can  achieve  given 


16  The  sortie  count  is  from  a  briefing  given  by  then-Lt  Gen  T.  Michael  Moseley  to  the 
CENTAF  Lessons  Learned  Conference,  Nellis  AFB,  Nevada,  July  18,  2003.  The  fighter 
force  structure  numbers,  which  are  plotted  in  Figure  6.1  in  Taylor  et  al.  (2002,  p.  98),  origi¬ 
nally  came  from  the  Air  Staff  (AF/XPPE).  In  FY  1994,  there  were  14,426  officers  in  11XX 
duty-coded  positions  (fighter  pilot).  In  FY  2001,  this  number  was  11,178,  for  a  reduction  of 
approximately  23  percent. 
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parts  funding,  manning  levels,  and  maintenance  experience  levels)  and 
the  flying-hour  funding  reductions  that  have  been  mandated  to  ease 
budgetary  disconnects  combine  to  limit  any  appreciable  increases  in 
available  flying  hours. 

Chapter  Three  will  examine  the  potential  for  improving  the  avail¬ 
able  training,  without  increasing  the  hours  flown,  by  using  advanced 
high-fidelity  simulation  facilities.  However,  we  will  first  examine  the 
causes  and  consequences  of  the  continuing  pressure  to  increase  man¬ 
ning  levels  and  decrease  experience  levels  in  operational  fighter  units. 


The  Production  of  New  Pilots  and  Overmanning  Issues 

During  the  post-Cold  War  drawdown  and  after  the  end  of  the  first 
Gulf  War,  pilot  training  was  cut  radically  rather  than  being  kept  at 
inventory  sustainment  levels.  Total  undergraduate  pilot  training  cohorts 
were  reduced  to  less  than  half  of  their  required  sustainment  levels,  and 
new  fighter  pilot  cohorts  were  reduced  even  more  aggressively,  with 
many  assignments  into  fighters  requiring  a  three-year  nonflying  period 
as  a  member  of  a  “pilot  bank”  before  formal  training  in  fighters  could 
begin.  These  low  pilot-production  rates  were  implemented  in  FY  1992 
and  continued  into  FY  1996,  creating  the  “pilot  bathtub”  (shortages  in 
pilot  inventories  relative  to  the  annual  cohorts  that  production  at  sus¬ 
tainment  levels  would  have  provided)  that  has  plagued  aircrew  manag¬ 
ers  for  over  a  decade. 

An  attempt  to  remedy  the  situation  was  made  at  a  four-star-rated 
summit  convened  by  the  Air  Force  Chief  of  Staff  in  1996,  which  set 
production  goals  for  new  fighter  pilots  at  370  pilots  per  year  begin¬ 
ning  in  FY  1997.  This  objective  could  not  be  met  and  was  eventually 
reduced  to  330  at  a  subsequent  summit  in  1999.  This  goal  has  also 
never  been  met  because  of  pipeline  production  constraints,  and  the 
number  of  experienced  fighter  pilots  has  continued  to  suffer  as  result. 

The  efforts  to  meet  these  goals,  however,  have  generated  prob¬ 
lems  with  overmanning  and  low  experience  levels  in  a  number  of  units 
because  the  flow  of  new  pilots  exceeded  the  absorption  capacity  of  the 
available  active  fighter  force  structure.  The  most  egregious  problems 
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occurred  in  the  A/OA-IO  unit  at  Pope  AFB,  North  Carolina,  in  FY 
2000,  but  F-15C  units  have  also  been  severely  stressed  throughout  the 
2000s.  When  units  are  overmanned  with  newly  trained  fighter  pilots, 
the  average  SCM  for  the  assigned  pilots  must  decrease  if  the  sortie 
resources  remain  constant.  This  means  that  inexperienced  pilots  will  fly 
even  less  because  additional  instructor  pilot  sorties  would  be  required 
to  train  the  excess  flow  of  incoming  pilots.17 

The  absorption  analysis  in  Taylor  et  al.  (2002)  carefully  examined 
these  issues,  but  the  dynamic  effect  of  the  ongoing  reduction  in  overall 
active  fighter  force  structure  raises  additional  concerns.  If  current  Air 
Force  leadership  is  sufficiently  intent  on  not  repeating  the  mistakes 
made  following  the  end  of  the  first  Gulf  War,  it  may  not  recognize 
the  damage  to  the  operational  training  environment  that  will  result  if 
existing  production  goals  are  not  reduced  to  match  the  force  structure 
reductions.  It  is  becoming  very  clear  that  the  Air  Force  will  not  be  able 
to  fly  its  way  out  of  the  impending  crisis  unless  it  can  use  more  air¬ 
frames  to  absorb  new  pilots  and/or  find  a  way  to  significantly  increase 
the  UTE  rates  flown  by  the  existing  airframes.  Indeed,  if  excessive 
numbers  of  new  pilots  are  sent  to  fly  a  continually  decreasing  number 
of  operational  aircraft,  those  who  leave  the  weapon  system  at  the  end 
of  an  initial  tour  will  have  less  of  an  opportunity  to  return  to  their 
primary  aircraft  following  an  intervening  tour.  Those  who  cannot  con¬ 
tinue  to  develop  their  operational  skills  during  a  second  flying  tour  in 
their  primary  aircraft  will  never  be  able  to  acquire  the  knowledge  or 
skills  that  are  required  to  fill  the  staff  and  operations  center  billets  that 
are  driving  the  high  production  quotas.  This,  of  course,  would  com¬ 
pletely  defeat  the  primary  purpose  of  training  higher  numbers  of  new 
pilots  in  the  first  place.18 

Our  historical  discussion  of  the  training  environment  has  con¬ 
firmed  that  fluctuations  in  training  effectiveness  are  not  a  new  phe- 


17  Chapter  Two  in  Taylor  et  al.  (2002)  discusses  the  so-called  “Pope  syndrome”  training 
issues. 

18  Taylor  et  al.  (2002)  documents  the  steady-state  absorption  results.  The  dynamic  issues 
specific  to  the  F-15C  were  briefed  to  the  Aircrew  Review  2005  attendees  on  December  12, 
2005.  The  latter  results  will  be  fully  documented  at  a  later  date. 
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nomenon  for  operational  fighter  units.  The  Air  Force  has  coped  with 
training  degradations  successfully  in  the  past  by  developing  new  pro¬ 
cesses  and,  when  necessary,  new  resources,  systems,  or  procedures  to 
improve  aircrew  training  and  management  methods.  The  issue  that  is 
new,  however,  and  the  one  that  will  make  the  Air  Force  reexamine  and 
refine  its  methods,  is  the  “decoupling”  of  the  absorption  and  develop¬ 
ment  processes  for  pilots:  Absorption  of  a  pilot  no  longer  ensures  that 
further  development  will  be  possible.  The  development  process  that 
enables  a  pilot  to  become  credentialed  as  a  flight  lead  or  instructor 
pilot  often  occurs  during  a  second  tour  in  his  or  her  primary  aircraft. 
Historically,  the  system  has  remained  sufficiently  balanced  to  ensure 
that  these  second  tours  were  always  available,  even  if  they  occurred 
after  an  intervening  assignment.  Thus,  once  a  pilot  was  absorbed,  his 
or  her  development  process  was  never  limited  by  the  lack  of  availability 
of  a  second  tour.  Ongoing  RAND  work  indicates,  however,  that  cer¬ 
tain  policy  options  could  drive  the  system  far  enough  out  of  balance 
to  severely  limit  opportunities  for  second  tours  and  continued  aircrew 
development  (Taylor,  2006). 

Both  the  quality  improvements  after  the  conflict  in  Southeast 
Asia  that  new  technology  (ACMI,  cockpit  video  recorders)  enabled 
and  the  quantity  improvements  that  new  aircraft  and  better  mainte¬ 
nance  enabled  contributed  to  the  development  of  an  outstanding  Air 
Force  fighter  capability.  Now  that  the  available  quantity  of  training  is 
declining,  an  important  question  is  whether  or  not  improvements  in 
training  quality  can  make  up,  at  least  in  part,  for  the  decline.  High- 
fidelity  simulation  systems  with  MTC  facilities  have  the  potential  to 
contribute  positively  to  the  training  process,  but  it  is  important  to  rec¬ 
ognize  that  they  will  not  resolve  the  training  crisis  that  will  result  from 
the  excessive  flows  of  new  pilots  into  the  operational  units  unless  these 
pilots  will  have  adequate  opportunities  to  continue  the  development 
of  essential  qualifications  in  subsequent  operational  assignments.  Fur¬ 
ther,  it  is  important  to  understand  whether  pilots  consider  the  types  of 
experience  available  from  MTC  facilities  to  be  equivalent  to  those  in 
the  aircraft.  It  is  also  important  to  understand  whether  current  RDTM 
measures  of  experience  are  still  valid,  both  for  unit  measures  of  effec¬ 
tiveness  and  for  an  individual  pilot’s  readiness  for  varying  positions  of 
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responsibility.  If  the  experience  assumption  that  was  fundamental  to 
the  development  of  the  RDTM  system  is  no  longer  valid,  other  mea¬ 
sures  need  to  be  examined,  because  aircrew  managers  will  require  new 
methods  for  identifying  and  tracking  the  continuing  developmental 
options  available  to  pilots  after  their  initial  operational  flying  opportu¬ 
nities.19  We  address  these  issues  in  the  next  two  chapters. 


19  The  Air  Force  has  recently  implemented  an  officer  development  system  that  requires  offi¬ 
cers  to  receive  periodic  review  and  feedback  from  an  appropriate  development  team  that 
includes  senior-level  members.  The  problem  at  this  point  for  rated  officers,  however,  is 
that  these  reviews  typically  begin  as  officers  become  eligible  for  promotion  to  the  grade  of 
major.  This  currently  occurs  after  they  normally  would  have  completed  a  second  operational 
assignment. 


CHAPTER  THREE 


High-Fidelity  Simulators  with  Mission  Training 
Centers 


High-fidelity  simulation  systems  coupled  with  MTCs  incorporate 
state-of-the-art  electronic  components,  including  high-resolution 
visual  graphics  and  compellingly  realistic  cockpit  components.1  In  air- 
superiority  units,  the  actual  cockpits  are  located  in  the  MTCs,  which 
also  incorporate  advanced  briefing  and  debriefing  facilities  and  ACMI- 
like  mission  histories  that  enable  pilots  to  construct  and  visually  decon¬ 
struct  all  the  engagements  accomplished  during  a  mission  and  to  use 
recreated  cockpit  displays  and  depictions  of  relative  aircraft  positions  to 
thoroughly  evaluate  kill  criteria,  weapon  effectiveness,  and  individual 
pilot  maneuvering  proficiencies  and  tactical  expertise.  These  systems 
also  incorporate  distributed  networks  that  enable  aircrews  to  include 
a  variety  of  potential  threat  environments  and  constructive  (computer¬ 
generated  and  fully  automated)  assets  and  adversaries  that  can  be  des¬ 
ignated  as  either  red  force  or  blue  force  resources.2  These  networks  will 
also  enable  other  operational  units,  as  well  as  appropriately  equipped 
and  networked  command-and-control  resources,  such  as  air  operations 
centers,  to  participate  simultaneously  in  employment  and  other  simula- 


1  This  brief  chapter  cannot  address  many  important  simulator-related  issues.  A  useful 
resource  is  Lee  (2005),  which  includes  chapters  on  visual  scene  simulation,  sound  effects  and 
communications  simulation,  flight  simulators  in  pilot  training  and  evaluation,  and  limita¬ 
tions  in  flight  simulator  design  and  use. 

2  It  is  useful  to  distinguish  between  constructive  resources,  which  are  computer  generated 
and  also  respond  and  react  during  the  session  in  accordance  with  their  programmed  soft¬ 
ware,  and  virtual  resources,  which  incorporate  a  person  in  the  loop  to  control  his  or  her 
responses  and  reactions. 
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tion  exercises.  These  networks  can  also  incorporate  input  data  from  the 
computer  systems  aboard  appropriately  linked  live  resources,  as  well  as 
simulators  and  other  linked  facilities.  The  ability  to  combine  construc¬ 
tive,  virtual,  and  live  resources  simultaneously  in  these  training  mis¬ 
sions  has  led  them  to  be  called  distributed  mission  operations  (DMO). 
Since  the  Air  Force  pioneered  the  development  of  these  systems,  many 
Air  Force  leaders  have  referred  to  the  simulator  systems  themselves  as 
DMO  systems.  In  this  monograph,  we  will  conform  to  the  terminol¬ 
ogy  the  Air  Force  currently  prefers  and  refer  to  them  as  high-fidelity  or 
DMO-capable  simulator  systems  because  the  full  DMO  capability  will 
not  be  realized  until  the  facilities  have  been  fully  funded  and  a  number 
of  additional  weapon  systems  have  been  linked  into  the  distributed 
network.3 

Earlier  simulation  training  systems  focused  on  training  skills  in 
instruments  and  navigation  and  on  skills  such  as  procedural  knowledge 
and  “switchology”  issues  that  apply  at  the  lower  end  of  the  building- 
block  model  (Figure  2.1).  The  advanced  simulator  systems,  however, 
also  make  it  possible  to  provide  simulator  training  in  skills  required  in 
the  higher  levels  of  the  training  spectrum,  such  as  ACT  and  LFE  sor¬ 
ties.  Thus,  these  new  resources  make  it  possible  to  develop  the  kinds 
of  skills  that  previously  required  the  participation  of  multiple  live  air¬ 
craft  and  often  a  great  deal  of  external  coordination,  cost,  and  effort 
to  conduct.4 

Air  Force  leadership  regarded  the  development  of  these  advanced 
simulator  systems  as  a  fundamental  component  of  a  training  trans¬ 
formation.  Indeed,  in  their  Senate  testimony  supporting  the  FY  2006 


3  Regarding  the  planned  scope  of  the  DMO  system,  Walker  (2004)  wrote. 

The  Air  Force  has  a  DMO  network  that  includes  four  aircraft  platforms — the  F-15C, 

F-16,  E-3  Airborne  Warning  and  Control  System  (AWACS),  and  the  E-8C  Joint  Sur¬ 
veillance  Targeting  Attack  Radar  System  (JSTARS).  New  platforms  are  slated  for  DMO 
participation,  including  the  F-15E,  R-135  Rivet  Joint  reconnaissance  aircraft,  B-l  and 
B-2  bombers  and  the  A- 10  ground  attack  aircraft.  The  F/A-22  simulators  at  Tyndall  Air 
Force  Base,  Fla.,  also  are  under  contract  to  become  DMO-compatible. 

4  For  the  most  recent  review  of  flying  simulation  research  specifically  related  to  combat 
training,  see  Bell  and  Waag  (1998). 
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budget,  the  acting  Secretary  of  the  Air  Force  and  the  then-Air  Force 
Chief  of  Staff  stated  the  following  (Jumper,  2005,  pp.  21-22): 

Distributed  Mission  Operations  (DMO)  is  the  cornerstone  for 
Air  Force  training  transformation.  It  is  a  readiness  initiative  to 
train  warfighters  as  they  expect  to  fight  using  simulation  and 
high-fidelity  architecture  to  link  training  at  dispersed  locations. 
DMO  will  reduce  travel  costs  and  operations  tempo  while  pro¬ 
viding  mission  rehearsal  in  an  operationally  realistic  environment 
to  maintain  combat  readiness  and  provide  support  to  operations. 

It  will  prepare  and  assess  Air  and  Space  Expeditionary  Forces  and 
prepare  AOC  [air  operations  center]  weapon  systems,  including 
Joint  Force  Air  Component  Commanders,  for  real-world  mis¬ 
sions.  As  an  integration  effort,  DMO  will  leverage  existing  and 
emerging  programs  and  technologies  to  fill  gaps  in  total  team 
training,  rehearsal,  and  operations  support. 

The  systems  are  designed  to  support  all  stages  of  flight  training, 
including  initial  qualification,  mission  qualification,  continuation, 
and  flight  lead  and  instructor  pilot  upgrade  training.  However,  MTC 
cockpits  in  fighter  units  do  not  include  motion  simulation  because  the 
technology  cannot  replicate  the  high  g-forces  that  pilots  experience  in 
modern  fighters.5  They  do  have  advantages  that  are  not  available  in  the 
aircraft,  however.  Earlier,  we  noted  four  characteristics  of  Red  Flag  that 
were  important  to  its  success: 

1.  a  relatively  unconstrained  volume  of  airspace  in  which  pilots 
could  maneuver  realistically 

2.  a  functioning  integrated  air  defense  system  based  on  the  Soviet 
model 

3.  the  opportunity  to  train  in  large  force  packages  similar  to  those 
planned  for  combat 


5  One  of  our  reviewers  noted  that  research  at  Armstrong  Laboratories  also  found  that  a  full- 
fidelity  cockpit  and  some  visual  cues  were  more  important  than  making  an  effort  to  include 
motion. 
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4.  a  comprehensive  debriefing  system  that  could  accurately  mea¬ 
sure  everything  from  an  individual  pilot’s  performance  to  an 
assessment  of  the  overall  tactical  plan. 

Each  of  these  elements  is  present  to  some  degree  in  the  high-fidelity 
simulators.  It  is  also  interesting  to  note  that  Desert  Pivot,  a  virtual  flag 
exercise  using  linked  DMO  resources,  was  initiated  in  October  2000 
at  Kirtland  AFB,  New  Mexico. 


The  Potential  Benefits  of  DMO-Capable  Systems 

Pilots  indicate  that  a  training  mission  in  an  MTC  cockpit  is  not  identi¬ 
cal  to  Hying  a  training  sortie  in  the  aircraft  but  that  the  fidelity  of  the 
training  is  far  greater  than  they  have  ever  experienced  in  older  simulator 
systems.  The  MTC  also  provides  cockpits  that  can  be  used  more  effi¬ 
ciently  for  training  than  can  actual  aircraft  cockpits.  For  example,  MTC 
training  is  not  as  susceptible  as  aircraft  sorties  are  to  cancellations  or  sig¬ 
nificant  modifications  to  the  original  mission  plan  because  of  weather  or 
mechanical  problems.  In  addition,  MTC  missions  do  not  involve  exten¬ 
sive  ground  operations,  departure  procedures,  transit  to  and  from  the 
range  or  military  operating  area,  and  air  traffic  control  issues  and  do  not 
require  recovery,  approach,  or  arrival  procedures.6  Furthermore,  the  abil¬ 
ity  to  quickly  reset  the  simulator  to  refly  an  engagement  or  initiate  a  new 
phase  of  the  training  profile  multiplies  the  quantity  of  training  that  can 
be  accomplished  per  unit  time  in  the  MTC  relative  to  that  of  a  similar 
training  profile  flown  as  a  live  flight.  A  live  mission  would  need  to  rejoin 
or  reform  the  aircraft  in  flight,  send  them  to  specified  starting  positions, 
and  set  things  up  again  from  scratch  to  refly  an  intercept  or  engagement. 
One  estimate  is  that  the  MTC  may  increase  the  quantity  of  training 
accomplished  per  unit  of  time  in  the  cockpit  by  a  4:1  ratio  over  the  quan¬ 
tity  available  on  a  live  mission  (BGI,  2002).  The  ACMI-like  capability  to 
replay  sorties  from  recorded  data  also  makes  mission  reconstructions  and 


6  Each  of  these  processes  can,  however,  be  accomplished  in  the  MTC  when  such  training  is 
required. 
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debriefings  more  efficient.  Less  time  is  spent  discussing  whether  some¬ 
thing  happened,  so  more  time  can  be  devoted  to  discussing  how  and  why 
things  happened  and  how  improvements  could  be  made  in  flying  similar 
missions  in  the  future. 

MTC  cockpits  can  be  regarded  as  additional  training  resources 
for  an  operational  unit.  Scheduling  this  training  effectively  requires  a 
degree  of  effort  similar  to  that  for  live  sorties,  and  the  training  missions 
consume  equivalent  amounts  of  personnel  resources  and  time.  This 
means  that  pilots  scheduled  for  MTC  training  are  not  available  for  live 
training  during  the  preparation,  execution,  and  reconstruction  phases 
of  the  MTC  training.  The  converse,  of  course,  is  also  true,  inasmuch 
as  pilots  scheduled  for  live  training  are  likewise  unavailable  for  MTC 
training  for  extended  periods  during  the  training  day.  Upgrading 
pilots  and  those  with  little  experience  require  exactly  the  same  amount 
of  supervision  from  flight  leads  and  instructor  pilots  on  MTC  mis¬ 
sions  as  they  would  require  on  a  live  sortie  with  a  similar  training  pro¬ 
file.  Therefore,  high-fidelity  simulators  do  not  create  additional  train¬ 
ing  time  in  a  pilot’s  day,  even  though  they  can  make  it  possible  to  use 
the  time  more  efficiently.  These  simulators  do  provide  extremely  useful 
training  resources  that  can  be  coordinated  with  pilots’  live  training 
schedules  and  significantly  enhance  their  development  of  additional 
skills  and  qualifications. 

The  new  simulator  systems  can  also  provide  training  to  cope  with 
emergency  circumstances,  such  as  battle  damage,  major  system  fail¬ 
ures,  and  other  serious  emergencies,  that  cannot  be  practiced  realisti¬ 
cally  during  live  training  in  an  aircraft.  As  one  of  many  examples  of 
the  value  of  simulator  training  to  cope  with  actual  in-flight  emergency 
situations,  Boeing  reports  that  a  pilot  discussing  how  he  landed  a  C-17 
after  his  number  two  engine  exploded  at  takeoff  from  Baghdad  (after 
taking  a  hit  from  enemy  ground  fire)  indicated  that  his  crew  had  prac¬ 
ticed  the  required  procedures  many  times  in  the  simulator  and  stated, 
“It  was  nothing  different  because  I  was  so  used  to  the  simulator.”  C-17 
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units  have  Federal  Aviation  Administration  Category  C+  simulator  sys¬ 
tems,  which  provide  as  high  a  fidelity  level  as  any  Air  Force  system.7 

MTCs  also  provide  opportunities  for  types  of  training  profiles 
that  are  extremely  difficult  and  costly  to  accomplish  in  a  live-fly  mode 
because  of  the  large  numbers  of  weapon  systems  and  other  resources 
that  are  required.  A  striking  example  of  these  options  is  the  opportu¬ 
nity  to  conduct  LFE  exercises.  Exercise  planners  must  invest  extensive 
effort  and  funds  to  task  and  assemble  a  realistic  composition  of  dis¬ 
parate  units  with  appropriate  mission  tasking  and  Designed  Opera¬ 
tional  Capability  (DOC)  statements  to  organize  and  execute  a  live- 
fly  LFE.8  Also,  the  participating  units  typically  must  devote  the  bulk 
of  their  training  resources  to  preparing  for  and  participating  in  the 
exercise,  thereby  preventing  upgrade  and  other  daily  training  opera¬ 
tions  for  a  fixed  period.  LFE  training  in  an  MTC  is  much  simpler  and 
less  expensive.  MTCs  provide  a  nearly  unlimited  supply  of  credible 
and  constructive  red  air  support  (an  MTC  can  manage  approximately 
200  entities  at  a  time  in  a  scenario),  and  the  mechanical  problems  that 
would  jeopardize  an  aircraft  exercise  of  this  magnitude  are  far  less 
likely  with  MTCs.  Finally,  the  MTC  provides  unlimited  constructive 
air  and  weapon-release  space  for  the  exercise.  DMO  offers  the  potential 
of  being  a  powerful  partner  of,  or  extension  to,  higher-end  training, 
such  as  traditional  Red  Flag,  or  LFE  exercises.9 

These  advanced  simulator  systems,  on  the  other  hand,  can  also 
complement  and  fully  support  more-basic  flying  training  needs.  For 
example,  the  MTC  can  be  used  to  practice  a  training  profile  prior  to 
flying  the  same  profile  in  an  aircraft,  reducing  the  probability  of  losing 
the  training  effectiveness  of  the  live  sortie  due  to  aircrew  mistakes  or 
planning  errors,  thereby  increasing  the  efficiency  of  the  available  air- 


7  See  Larock  (2005b).  See  also  Larock  (2005a)  for  additional  information  on  DMO  capa¬ 
bilities  in  the  F-15C  and  other  aircraft. 

8  DOC  statements  “provide  a  narrative  description  of  the  unit’s  wartime  mission(s).”  See 
AFI  10-201  (2006,  para.  1.7). 

9  Maj  Gen  Teresa  Marne  Peterson,  the  former  Air  Staff  Director  of  Operations  and  Train¬ 
ing  (AF/A30),  said  that  DMO  exercises  are  available  “at  a  fraction  of  the  cost  of  getting 
everyone  together  at  the  same  range”  (quoted  in  Hebert,  2004,  p.  41). 
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craft  training  time.  When  live-fly  sortie  resources  are  in  short  supply, 
as  has  been  the  case — especially  for  inexperienced  wingmen — in 
F-15C  units  since  FY  2000,  training  schedules  can  be  coordinated  so 
that  the  limited  number  of  live  sorties  can  be  used  more  efficiently. 
For  example,  Bennett  and  Crane  (2002)  reported  that  spin-up  training 
using  MTC  reduced  noneffective  sorties  at  USAF  Weapons  School  by 
12  percent.10  We  will  return  to  this  notion  later  because  we  feel  that  it 
requires  additional  analysis  to  exploit  the  opportunities  fully. 

Units  can  also  rehearse  missions  in  specific  theaters  where  they 
may  expect  to  be  tasked  to  conduct  combat  operations.  This  means 
that  they  can  fly  against  constructive,  integrated  air-defense  systems 
that  are  essentially  identical  (within  available  intelligence  estimates  and 
computer  programming  limitations)  to  those  they  will  encounter  in 
wartime  mission  tasking.  Force  package  size  and  the  types  of  partici¬ 
pation  are  nearly  unlimited,  and  this  is  valuable  on  the  higher  end  of 
the  training  spectrum  because  it  enables  crews  to  practice  the  complex 
integration  and  communication  issues  of  operating  in  a  large,  diverse 
package  of  combat  aircraft  in  constant  communication  with  combat 
operations  centers.* 11  Indeed,  following  the  first  night  of  combat  opera¬ 
tions  in  Operation  Iraqi  Freedom,  a  highly  experienced  F-16  defense 
suppression  instructor  pilot  and  Weapons  School  graduate  made  the 
following  statement: 

I  was  downtown  at  the  start  of  tonight’s  activities  and  got  to  launch 

the  first  FIARM  [Fligh-Speed  Antiradiation  Missile]  of  Opera¬ 
tion  Iraqi  Freedom.  .  .  .  the  location  of  my  flight  and  the  tactics 


10  See  also  Crane,  Robbins,  and  Bennett  (2000).  Spin-up  training  refers  to  training  con¬ 
ducted  to  prepare  pilots  for  deployment  in  a  specific  theater. 

11  Without  exception,  the  pilots  we  interviewed  acknowledged  the  usefulness  of  simulator 
training  and  universally  praised  the  advanced  simulators  with  MTC  capability — especially 
in  comparison  to  the  previous  generation  of  simulators.  Many,  however,  would  immediately 
follow  positive  comments  with  concerns  and  issues  comparing  simulator  training  to  live 
training.  These  concerns  dealt  primarily  with  the  “airmanship  skills”  that  are  discussed  in 
the  next  section. 
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employed  were  EXACTLY  like  we  were  practicing  in  the  F-16 
MTC  at  Shaw  before  we  left.  .  .  .  talk  about  Mission  Rehearsal !12 


Shortcomings  of  Simulator  Systems 

Inherent  Lack  of  Fidelity 

Even  the  most  advanced  of  modern  simulators  has  an  inherent  lack  of 
fidelity.  Very  knowledgeable  engineers  and  pilots  who  met  with  our 
research  team  members  stressed  this  fact  during  site  visits  to  simulator 
research  and  development  centers.13  The  most  obvious  reaction  that  is 
difficult  to  simulate  is  the  psychological  stress  that  can  be  caused  by 
anticipation  of  the  catastrophic  consequences  of  serious  in-flight  errors 
or  lapses  in  judgment.  We  heard  from  several  sources  that  pilots  always 
realize  that  they  will  walk  away  from  their  simulator  cockpit  because 
no  one  has  ever  died  from  errors  made  during  simulator  training. 
The  same  sense  of  security  is  rarely  present  in  flight,  especially  during 
combat  operations.  These  sources  do  confirm  that  realistic  simulator 
training  can  certainly  enhance  one’s  ability  to  cope  with  and  overcome 
many  of  these  anxieties.  Certain  issues,  however,  such  as  the  stark  reali¬ 
ties  of  the  catastrophic  loss  of  a  flight  member  in  an  explosion  or  crash, 
cannot  be  simulated  very  effectively.14 


12  The  quote  is  taken  from  a  briefing  by  the  ACC/DO  (now  ACC/A3)  to  the  2003  Combat 
and  Mobility  Air  Forces  Commanders’  Conference,  Nellis  AFB,  October  2003;  capitaliza¬ 
tion  and  emphasis  are  as  in  the  original.  HARM  is  the  primary  munition  against  many 
enemy  air  defense  systems.  It  should  also  be  noted  that  the  Iraqi  air  defense  system,  which 
remained  essentially  intact  at  the  beginning  of  Iraqi  Freedom,  was  the  primary  target  during 
the  first  night’s  operations.  “Downtown”  refers,  of  course,  to  downtown  Baghdad,  where  the 
Iraqi  air  defenses  were  concentrated  and  should  have  been  the  most  effective. 

13  The  sites  visited  included  the  Integrated  Defense  Systems  Division  of  the  Boeing  Com¬ 
pany,  St.  Louis,  Missouri;  the  NASA  Ames  Research  Center,  Moffett  Field,  California;  and 
AFRL’s  Warfighter  Readiness  Research  Division,  Mesa,  Arizona. 

14  One  reviewer  notes  that,  in  the  current  environment,  simulator  training  sometimes  has 
better  “fidelity”  to  wartime  training  than  do  live  sorties.  Since  F-15s  and  F-16s  flying  peace¬ 
time  missions  were  configured  differently  from  wartime  missions,  researchers  at  Armstrong 
Laboratories  found  that  “negative”  training  was  occurring  in  the  air  that  could  be  overcome 
with  early  design  of  simulators  with  a  full-fidelity  cockpit. 
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Another  well-known  limitation  in  simulator  fidelity  affects  the 
skills  required  to  visually  acquire  and  identify  airborne  targets  and 
ground  references.  The  computer  graphics  and  visual  display  systems 
associated  with  simulator  systems  are  continually  improving,  but  they 
still  lack  lifelike  resolution.  These  visual  limitations  result  primarily 
from  technical  limitations  in  the  size  and  number  of  pixels  that  can  be 
generated  and  displayed  within  the  optical  systems,  but  they  also  relate 
to  the  requirement  to  display  the  visual  features  on  two-dimensional 
screens  within  the  MTC  cockpit.  This  lack  of  three-dimensional  cues 
is  especially  notable  during  high-speed  operations  at  very  low  altitudes 
(e.g.,  within  300  feet  of  the  ground).  These  operations  are  much  less 
important  for  air-superiority  units  and  have  become  less  so  for  multi¬ 
role  fighter  units  employing  standoff  weapons,  but  they  could  become 
critical  once  again  for  these  units  as  tactics  evolve. 

The  final  limitation  in  simulator  fidelity  for  fighter  units  is  the 
inability  to  replicate  the  g-forces  exerted  on  the  pilots  during  normal 
fighter  operations.  These  forces  can  reach  9  g,  which  means  that  objects 
in  the  cockpit,  including  the  internal  organs  and  other  body  parts  of  the 
pilot,  are  subjected  to  nine  times  the  forces  normally  exerted  in  straight 
and  level  flight.  It  can  be  exhausting  to  maintain  the  blood  supply  to 
the  pilot’s  brain  and  eyes,  neither  of  which  will  function  properly  when 
the  oxygen  supply  carried  in  the  blood  is  interrupted. 

Airmanship 

Knowledgeable  pilots  and  simulator  experts  categorize  many  of  the  pilot 
skills  affected  by  the  fidelity  limitations  as  airmanship  skills.  Many  of 
the  same  experts  agree  that  improving  airmanship  skills  is  one  of  the 
most  important  elements  of  developing  overall  pilot  skills,  increasing 
pilots’  qualifications  and  preparing  them  for  greater  responsibilities. 
The  implication  may  be  that  more  highly  qualified  pilots  can  maintain 
their  skills  more  reliably  using  advanced  simulator  training  than  could 
newer,  less-experienced  pilots  whose  airmanship  skills  are  not  yet  as 
highly  developed.  This  concept  deserves  more  careful  examination. 

The  Air  Force  has  long  believed  that  the  flying  skills  of  less- 
experienced  pilots  are  more  perishable  because  maintaining  these  skills 
consistently  requires  more  training  and  practice  for  them  than  for  more 
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experienced  pilots.  Indeed,  the  Ready  Aircrew  Program  tasking  mes¬ 
sages  for  all  fighters  require  fewer  sorties  per  training  cycle  for  pilots 
who  meet  the  500-hour  experience  definition  than  for  those  who  do 
not.  The  training  models  we  developed  for  operational  units  in  our 
earlier  work  partially  confirmed  this  point,  but  it  also  appeared  to  us 
that  pilots  who  had  recently  qualified  for  additional  responsibilities, 
such  as  new  flight  leads,  needed  almost  as  much  training  per  unit  time 
to  maintain  and  develop  the  new  skills  required  to  lead  flights  as  did 
the  inexperienced  wingmen.  Similar  training  needs  also  apply  to  new 
instructor  pilots.  This  means  that  most  pilots  who  actually  require  less 
training  are  the  experienced  instructor  pilots.  These  pilots  would  have 
to  be  considered  highly  experienced  pilots  in  any  meaningful  distribu¬ 
tion  of  experience  based  on  pilot  qualifications  and  the  skills  required 
to  perform  the  corresponding  duties.15 

To  relate  these  issues  to  simulator  training  in  advanced  facilities, 
it  will  be  instructive  to  look  at  the  training  policies  of  the  major  air¬ 
lines.  It  is  worth  noting  that  the  major  airlines  never  use  aircraft  equip¬ 
ment  solely  to  provide  live  training  for  their  pilots.  The  very  first  live 
flight  a  newly  hired  pilot  makes  will  be  a  regularly  scheduled  flight 
carrying  revenue  passengers  or  cargo.  All  the  initial  training  required 
to  prepare  a  new  pilot  for  a  first  flight  takes  place  in  high-fidelity  simu¬ 
lators.  Indeed,  all  the  upgrade  training  required  for  airline  pilots  to 
change  crew  positions  or  transition  to  new  types  of  equipment  is  also 
conducted  in  high-fidelity  simulators.  Several  features  of  this  training 
contribute  to  the  airlines’  ability  to  avoid  the  costs  that  would  result 
from  using  aircraft  solely  for  crew  training,  but  the  primary  factor  that 
distinguishes  airline  and  Air  Force  training  needs  is  the  experience 
levels  of  the  new  pilots.  We  visited  several  major  airlines  in  the  late 
1990s,  when  they  were  still  aggressively  hiring  new  pilots,  and  discov¬ 
ered  that  they  were  hiring  former  military  pilots  with  at  least  1,500  to 
3,000  flying  hours,  depending  on  the  military  weapon  system  a  pilot 
had  flown,  or  civilian  pilots  with  at  least  4,500  flying  hours.  Simulator 


15  It  is  perhaps  ironic  that  the  highly  experienced  instructor  pilots,  who  require  less  training 
per  month  to  maintain  their  own  flying  skills,  are  precisely  the  ones  who  will  fly  more  sorties 
per  month  in  a  unit  overmanned  with  new  pilots,  to  supervise  their  upgrade  requirements. 
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hours  were  neither  counted  nor  credited  in  the  hiring  process.  It  should 
be  noted  that  these  new  hires  are  very  highly  experienced  relative  to  the 
operational  fighter  world  in  the  Air  Force.  Pilots  with  this  much  flying 
experience  would  be  unlikely  to  require  much  training  in  the  basic  air¬ 
manship  skills  that  simulators  may  not  be  able  to  provide  adequately. 
Indeed,  to  meet  the  commercial  1,500-hour  minimum  for  hiring  mili¬ 
tary  fighter  pilots,  they  would  have  to  complete  at  least  three  typical 
flying  assignments  in  their  primary  aircraft.  In  an  operational  fighter 
wing,  pilots  with  that  level  of  background  and  experience  would  be 
available  only  for  very  senior  supervisory  and  command  positions.16 

At  the  other  end  of  the  qualification  hierarchy,  additional  RAND 
research  confirmed  that  airmanship  issues  were  precisely  the  reason  the 
Air  Education  and  Training  Command  has  resisted  replacing  live  sor¬ 
ties  with  simulator  training  in  the  formal  training  unit  programs  that 
initially  qualify  new  fighter  pilots.  The  airmanship  factors  were  identi¬ 
fied  as  follows:  “Pulling  g’s,  the  fear  of  death  from  making  a  mistake, 
heat,  radio  traffic,  and  equipment  failures  were  all  described  as  difficult 
or  impossible  to  simulate”  (Ausink  et  ah,  2005,  p.  64). 

We  introduce  this  information  because  it  supports  the  assertion 
that  highly  experienced  pilots,  whose  basic  airmanship  skills  are  fully 
developed,  have  the  potential  to  rely  more  heavily  on  appropriate  simu¬ 
lator  training  to  maintain  or  develop  primary  flying  skills  than  those 
with  little  experience,  who  must  continually  strive  to  develop  their  air¬ 
manship  and  other  skills.  A  related  concept,  raised  by  the  experts  at 
Boeing,  is  that  the  live  sorties  that  are  available  should  be  carefully 
managed,  especially  when  they  are  in  short  supply,  to  ensure  that  newly 
assigned  pilots  continue  to  develop  all  skills  required,  including  those 
that  are  less  effectively  inculcated  by  advanced  simulators.  This  requires 
the  units  to  use  appropriately  coordinated  and  balanced  live  and  simu¬ 
lator  training  programs. 

The  Air  Force  Research  Laboratory  (AFRL)  and  ACC  have  spon¬ 
sored  a  considerable  body  of  research  addressing  these  issues.  Their 


16  The  visits  to  major  airlines  supported  our  research  on  the  effect  of  airline  hiring  on  Air 
Force  pilot  retention,  which  is  documented  in  Taylor,  Moore,  and  Roll  (2000).  The  airlines 
visited  included  American,  Delta,  and  Southwest. 
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studies  address  the  mission-essential  competencies  required  for  tacti¬ 
cal  flying  and  the  relationship  between  the  operational  training  envi¬ 
ronment  and  the  combat  circumstances  within  which  the  pilots  are 
training  to  employ  their  tactical  flying  skills.  This  research  provides 
a  foundation  on  which  to  base  training  directives  that  incorporate 
a  more-efficient  mix  of  live  and  simulator  training.  (See  Colegrove, 
2005,  and  Colegrove  and  Bennett,  2005.) 


The  Role  of  DMO-Capable  Simulators  in  Operational 
Fighter  Units 

Our  interviews  and  discussions  with  simulator  experts  as  well  as  our 
visits  with  operational  supervisors,  instructor  pilots,  and  our  subse¬ 
quent  discussions  with  concerned  staff  officers  all  confirm  that  there 
is  a  great  deal  of  value  in  the  training  that  pilots  receive  in  high- 
fidelity  simulator  systems.17  Indeed,  there  was  universal  acceptance  of 
the  fact  that,  if  the  simulator  training  could  be  scheduled  to  comple¬ 
ment  and  support  an  appropriate  live  training  schedule,  the  live  train¬ 
ing  that  is  available  could  be  quite  a  bit  more  effective  in  develop¬ 
ing  and  maintaining  required  skills  than  would  a  similar  amount  of 
live  training  that  was  not  augmented  with  the  simulator  training.  This 
raises  a  related  question  about  how  (or  whether)  the  extra  training  that 
can  be  accomplished  per  flying  hour  should  be  credited  to  the  pilots. 

Our  analysis  has  led  us  to  the  conclusion  that  DMO-capable  sim¬ 
ulator  training  should  indeed  be  credited  to  the  pilots  in  an  appropriate 
manner  and  that  this  training  should  be  included  in  training  require¬ 
ments  so  that  unit  supervisors  can  thoroughly  track  and  evaluate  the 
training,  rather  than  rely  solely  on  an  undocumented  system  consisting 
of  subjective  judgments.  This  is  the  only  option  we  see  that  can  ensure 
the  coordination  and  integration  of  the  live  training  the  pilots  receive 
with  the  available  simulator  training.  This  process  should  support  an 


17  The  staff  offices  visited  included  the  operations  and  training  shops  at  the  Air  Staff  (AF / 
A30T),  Air  Combat  Command  Operations  and  Training  (ACC/A3T),  and  Air  Mobility 
Command  Operations  and  Training  (AMC/A3T). 


High-Fidelity  Simulators  with  Mission  Training  Centers  33 


ongoing  effort  to  improve  the  overall  training  quality  available  in  the 
operational  units. 

Our  preliminary  dynamic  estimates  indicate  that  current  pro¬ 
grammed  pilot-production  levels,  coupled  with  the  programmed  reduc¬ 
tions  in  the  F-15C  force  structure  (and  consequently  in  the  numbers 
of  units),  will  mean  that  at  least  80  percent  of  the  API-1  billet  autho¬ 
rizations  in  these  units  by  FY  2011  will  be  filled  with  first-tour  pilots. 
This  would  preclude  the  continued  development  of  most  of  these  pilots 
during  a  second  tour  in  their  primary  aircraft,  and  our  final  chapter 
will  address  the  fact  that  these  tours  will  remain  absolutely  essential  for 
pilots  to  develop  into  qualified  supervisors  and  staff  officers.18 

The  potential  benefits  of  greater  use  of  high-fidelity  simulator 
systems  thus  outweigh  well-known  limitations  of  training  in  these  sys¬ 
tems.  The  Air  Force  should  therefore  formalize  methods  of  counting 
the  time  in  these  systems  to  help  measure  experience. 


Additional  Concerns  and  Caveats 

Funding 

The  advanced  simulator  facilities  have  funding  issues.  Currently,  no 
operational  fighter,  including  the  F-15C,  is  fully  supported  by  DMO- 
capable  simulator  systems.  The  F-15C  is  indeed  the  most  robust  system, 
but  one  squadron  has  no  advanced  simulator  facilities,  and  another  has 
only  a  two-cockpit  MTC.19  The  Block  50  version  of  the  F-16C  is  cur¬ 
rently  the  only  other  fighter  with  an  advanced  simulator  capability,  and 
the  remaining  F-16Cs,  as  well  as  the  F-15Es  and  A/OA-lOs,  all  have 
simulator  funding  issues  that  need  to  be  resolved.  Even  projected  F-22A 
units  have  simulator  funding  issues.  For  the  concept  of  coordinated  and 


18  This  problem  is  dynamic,  so  it  is  difficult  to  quantify  accurately.  It  should,  however,  be 
easy  to  see  that  continuing  force-structure  reductions  during  the  nominal  four  years  after 
a  pilot’s  first  tour  in  a  given  aircraft  type  will  mean  that  even  fewer  of  the  aircraft  will  be 
available,  making  it  more  difficult  for  that  pilot  to  return  for  a  second  tour.  We  should  also 
emphasize  that  the  80-percent-fill  factor  could  be  considerably  lower  than  the  actual  value. 

19  When  air-superiority  tactics  are  based  on  a  fundamental  four-ship  employment  unit, 
two-ship  training  can  be  far  from  adequate. 
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integrated  training  programs  to  be  efficient,  all  units  must  have  essen¬ 
tially  identical  simulator  systems  that  are  fully  functional  throughout  the 
fleet,  or  such  systems  must  be  readily  available  to  all  units. 

An  additional  funding  issue  that  affects  simulator  effectiveness  is 
the  requirement  to  ensure  that  the  software  programs  in  the  simulator 
remain  consistent  with  those  in  the  aircraft.  The  Air  Force  had  to  remove 
the  previous-generation  F-16C  simulators  from  operational  units  in  the 
1990s  because  funding  issues  precluded  keeping  that  system’s  opera¬ 
tional  flight  program  software  aligned  with  changes  that  were  being 
made  in  the  aircraft’s  software.  Simulators  with  outdated  software  can 
easily  result  in  negative  training,  especially  for  new  pilots. 

The  final  funding  issues  are  current  and  projected  reductions  in 
flying-hour  funding.  A  reduction  in  F-15C  flying  hours  was  required 
to  free  funding  for  additional  DMO-capable  systems  for  other  MDSs. 
Subsequent  flying-hour  cuts  will  be  required  for  the  Air  Force  to  resolve 
ongoing  budgetary  disconnects  in  the  FY  2008  Program  Objective 
Memorandum  because  options  for  reducing  discretionary  spending  are 
extremely  limited.  Some  of  these  reductions  may  be  offset  by  contin¬ 
gency  flying  hours,  but  the  availability  and  training  value  of  these  will 
need  to  be  carefully  examined  as  the  Air  Force  develops  its  integrated 
training  programs.20 

Scheduling  and  Supervision 

Our  final  concerns  deal  with  the  scheduling  and  supervision  of  simula¬ 
tor  training.  As  previously  discussed,  for  most  training  profiles,  inex¬ 
perienced  pilots  and  those  with  lower  qualification  levels  require  the 
same  level  of  supervision  in  the  MTC  as  they  do  in  the  aircraft.  When 
flight  leads  and  instructor  pilots  are  in  high  demand  and  short  supply 
in  a  unit,  they  will  supervise  far  more  training  than  they  individu¬ 
ally  require.  The  advanced  simulators  are  operated  under  contract  at 
all  Air  Force  units,  but  policies  on  whether  contractor  personnel  can 
provide  the  required  training  supervision,  by  acting  as  instructor  pilots 
or  flight  leads  in  the  MTCs,  differ  by  location.  These  policies  need  to 
be  evaluated  and  standardized  if  the  Air  Force  is  going  to  build  fully 


20  Erwin  (2000)  also  discusses  the  issues  associated  with  outdated  software  in  simulators. 
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effective  integrated  live  and  simulator  training  programs.  If  the  objec¬ 
tive  of  consistent,  effective  training  across  all  the  operational  units  is 
to  be  achieved,  overall  training  effectiveness  cannot  be  allowed  to  vary 
significantly  from  one  location  to  another. 

Our  concerns  embrace  both  ends  of  the  policy  spectrum  on  this 
issue.  If  contractor  personnel  are  going  to  supervise  actual  training, 
the  Air  Force  needs  to  ensure  that  their  supervision  skills  meet  accept¬ 
able  quality  standards  and  must  also  evaluate  the  sustainability  of  con¬ 
tractor  sources  of  adequately  qualified  training  supervisors.  If  this  is 
the  preferred  option,  the  Air  Force  must  also  establish  policies  that 
will  ensure  that  contractor  training  supervisors  can  remain  current  on 
evolving  tactics  and  munitions  capabilities.  To  the  best  of  our  knowl¬ 
edge,  none  of  these  issues  has  been  adequately  addressed. 

If,  on  the  other  hand,  the  preferred  policy  is  to  use  active-duty 
supervisors  in  these  training  facilities,  the  Air  Force  must  thoroughly 
analyze  the  scheduling  issues  that  could  result  and  determine  the  avail¬ 
ability  of  qualified  training  supervisors  who  would  also  be  required  to 
supervise  all  the  unit’s  live  training.  The  same  issues  that  prevent  inex¬ 
perienced  wingmen  from  getting  on  the  flying  schedule  could  easily 
limit  their  simulator  training  as  well.  In  the  operational  world  that  we 
envision  for  the  near  term,  it  may  not  be  feasible  for  limited  numbers  of 
qualified  pilots  to  be  available  to  provide  all  the  supervision  that  would 
be  required  for  an  integrated  live  and  simulator  training  program 
and  still  remain  within  crew-rest  constraints.  If  so,  the  Air  Force  will 
need  to  examine  alternative  sources  for  simulator  training  supervision. 
Simply  changing  the  definition  of  an  experienced  pilot  and  declaring 
the  operational  training  problem  solved — without  developing  the  effi¬ 
cient  integrated  training  program  and  the  aircrew  management  initia¬ 
tives  required  to  deal  with  the  impending  issues — could  have  serious 
training  and  operational  consequences  in  the  future. 

Impending  Issues 

Chapter  Four  will  document  the  training  environment  evaluations  that 
we  conducted  and  incorporate  the  results  of  our  interviews  and  surveys 
from  October  2002  through  October  2003.  These  evaluations  inspired 
our  initial  concerns  about  the  adequacy  of  the  500-hour  experience 
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definition  for  assessing  a  unit’s  health  when  determining  whether  the 
unit  has  enough  pilots  who  are  qualified  to  manage  its  training  needs. 
These  results  also  confirmed  for  us  the  advantages  of  coordinating  and 
integrating  the  live  and  simulated  training  profiles  available  to  pilots  in 
a  unit.  We  hope  that  the  discussion  that  follows  can  also  help  to  inform 
the  process  required  to  develop  the  integrated  training  program  that 
we  advocate. 


CHAPTER  FOUR 

Experience  and  Development  Surveys 


No  objective,  universally  agreed-on  method  exists  for  measuring  a 
pilot’s  actual  level  of  experience.1  For  one  thing,  a  pilot’s  experience 
may  be  meaningful  only  in  an  appropriate  context — for  example,  an 
experienced  instructor  pilot  is  clearly  distinct  from  an  experienced  pilot. 
Historically,  even  pilots  who  meet  the  500-hour  RDTM  definition  do 
not  all  reach  each  the  same  qualification  level  at  the  same  time. 

We  therefore  needed  to  find  out  what  aspects  of  a  pilot’s  training 
other  than  flying  hours — such  as  upgrade  level  and  training  experi¬ 
ence  in  DMO-capable  simulator  systems — should  be  counted  in  the 
determination  of  a  pilot’s  experience  level.2  This  led  us  to  ask  how  pilots 
themselves  determine  when  someone  is  “experienced,”  whether  they 
expected  a  potential  combat  leader  and  a  staff  leader  to  have  different 
kinds  and  amounts  of  experience,  and  whether  they  found  DMO  train¬ 
ing  an  acceptable  form  of  experience.  To  answer  to  these  questions,  we 
surveyed  experts  in  fighter-pilot  training  to  determine  how  well  the 
current  RDTM  flying-hour  definition  of  experience  corresponds  with 
how  they  evaluate  a  pilot’s  level  of  experience. 


1  The  apparently  objective  RDTM  system  for  measuring  experience  in  terms  of  flying 
hours  is  actually  based  on  a  subjective  judgment  of  training  experts  about  the  criteria  that 
should  define  an  experienced  pilot.  We  have  spoken  with  several  training  experts  who  now 
believe  these  criteria  do  not  give  a  good  indication  of  the  experience  levels  of  the  current  crop 
of  fighter  pilots,  illustrating  the  subjective  nature  of  these  criteria. 

2  Upgrade  level  refers  to  the  position  a  pilot  is  qualified  to  fly  in  a  formation  and  whether  the 
pilot  is  a  qualified  instructor  pilot.  The  levels  of  upgrade  are  wingman ,  two-ship  flight  lead, 
four-ship  flight  lead,  and  mission  commander. 
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The  survey  used  policy-capturing  methods  to  determine  the  bases 
for  expert  pilots’  judgments  of  other  pilots’  combat  readiness  and  their 
readiness  for  staff  positions  (Anderson,  1981;  Kline  and  Sulsky,  1995). 
These  methods  use  statistical  models  to  discover,  or  capture ,  how  experts 
combine  information  in  forming  judgments.  The  combination  rule  that 
is  the  basis  of  these  judgments  represents  the  experts’  policy  as  applied  in 
making  the  judgment  at  hand.  These  methods  are  particularly  well  suited 
to  the  current  research,  in  which  one  possible  basis  of  an  expert’s  judg¬ 
ment  of  a  pilot’s  experience  is  the  amount  of  training  the  pilot  received  in 
DMO-capable  simulators.  Fighter  pilots  may  be  reluctant  to  endorse  the 
use  of  high-fidelity  simulators  in  training  if  doing  so  could  be  interpreted 
as  willingness  to  trade  aircraft  flying  time  for  simulation  training  time. 
However,  using  policy-capturing  techniques,  experts  are  never  explicitly 
asked  to  endorse  such  a  trade-off,  so  their  judgments  are  less  likely  to 
be  tainted  by  this  concern.  With  the  surveys  described  in  this  chapter, 
policy-capturing  methods  were  used  to  deduce  the  attitudes  of  training 
experts  about  the  role  of  high-fidelity  simulator  systems  and  other  train¬ 
ing  events  in  the  development  of  experienced  pilots. 

We  conducted  two  surveys  over  a  one-year  period,  October  2002  to 
October  2003.  The  first  survey  was  given  to  F-16  training  experts  at  Hill 
AFB  and  represented  a  test  of  the  policy-capturing  methodology.  The 
second  survey,  conducted  with  F-15C  training  experts  at  Langley  AFB, 
Virginia,  and  Eglin  AFB,  Florida,  reflected  lessons  learned  from  the  first 
survey  and  included  questions  about  training  in  DMO-capable  simula¬ 
tor  systems.  DMO  training  was  already  part  of  the  training  curriculum 
at  Langley.  Results  of  the  F-16  survey  were  used  primarily  to  refine  the 
survey  structure  and  questions,  so  we  will  focus  on  the  F-15  survey.3 


The  F-15  Survey 

Goals 

Using  lessons  learned  from  the  first  survey,  we  designed  the  second  to 
give  us  a  better  chance  of  learning  the  informational  basis  for  expert 


3  The  appendix  details  the  F-16  survey  and  its  results. 
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judgments  of  pilots’  experience  in  terms  of  readiness  for  a  staff  position 
and  readiness  for  combat. 

The  Survey 

At  the  core  of  the  survey  were  20  fictional  pilot  profiles,  provided  in 
an  Excel  file.  This  popular  file  format  allowed  email  portability  and 
permitted  the  respondent  to  set  the  survey  aside  to  be  continued  later, 
if  necessary.  It  also  gave  the  respondent  easy  access  to  the  instructions 
while  taking  the  survey. 

The  Respondent  Pool 

We  sent  the  survey  to  40  F-15  training  experts  at  Langley  and  Eglin 
AFBs.  Of  these,  20  completed  and  returned  the  survey,  for  a  return 
rate  of  50  percent.  These  individuals  constituted  a  fairly  representa¬ 
tive  sample  of  unit  training  experts.  The  number  of  respondents  from 
the  different  positions  sampled  was  nearly  proportional  to  the  total 
number  of  people  in  those  positions. 

Methods 

We  set  up  the  program  to  generate  detailed  fictional  pilot  profiles  ran¬ 
domly.  Each  expert  evaluated  a  unique  set  of  20  different  profiles.  We 
tried  to  minimize  the  amount  of  guesswork  about  which  details  of  each 
fictional  pilot’s  background  and  training  might  influence  the  expert 
judgments  of  the  pilot’s  experience. 

Figure  4.1  is  an  example  of  one  such  fictional  pilot  profile  as  a 
respondent  would  see  it.4  The  profile  provides  information  about 
a  pilot’s  overall  training  history  and  details  about  assignments  before 
entering  the  current  operational  unit  (see  the  upper  left  corner  of  the 
figure).  The  fictional  pilots’  experience  ranged  from  having  completed 
only  undergraduate  pilot  training  to  having  flown  a  different  airframe 
and  having  spent  time  in  a  staff  liaison  tour.  This  information  about  a 
pilot’s  experience  turned  out  to  be  a  major  determinant  of  expert  judg¬ 
ments  regarding  the  pilot’s  readiness  for  a  staff  position. 


4  During  their  service  at  RAND  as  Air  Force  Fellows,  knowledgeable  Air  Force  fighter 
pilots  reviewed  the  pilot  surveys  and  developed  data  for  the  profiles. 


Figure  4.1 

Example  Pilot  Profile  Display  from  the  Survey  of  F-15  Experts 
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Information  about  the  pilot’s  current  status  appears  in  the  upper 
right  corner  of  the  profile  (as  shown  in  Figure  4.1)  and  included  the 
pilot’s  current  upgrade  level  (wingman,  two-ship  flight  lead,  four-ship 
flight  lead,  instructor  pilot,  or  mission  commander),  primary  aircraft 
flying  hours,  total  flying  hours,  and  months  in  the  current  unit.  The 
lower  portion  of  the  profile  provides  the  pilot’s  detailed  six-month 
training  history,  with  separate  sections  for  aircraft  and  simulator  sor¬ 
ties.5  The  display  shows  the  number  of  sorties  flown  each  month  and 
the  position  (mission  commander,  four-ship  flight  lead,  two-ship  flight 
lead  or  wingman)  in  which  the  sorties  were  flown.  The  display  shows 
eight  types  of  aircraft  sorties  that  represent  the  range  of  sorties  flown 
in  fighter  training:  BFM,  ACM,  two-  and  four-ship  air  ACT  (two-ship 
ACT,  four-ship  ACT),  Red  Flag,  other  LFE  sorties,  Red  Air,  and  AEF 
sorties.  The  display  also  shows  four  types  of  high-fidelity  simulator  sor¬ 
ties:  ACM,  two-  and  four-ship  ACT  (“ACT,  2vX”  and  “ACT,  4vX”  in 
Figure  4.1),  and  LFE. 

We  asked  the  experts  to  make  two  judgments  about  pilot  experience 
level,  the  first  in  terms  of  readiness  for  a  staff  assignment  and  the  second 
in  terms  of  readiness  for  combat.  They  were  asked  to  judge  readiness  for  a 
staff  assignment  by  estimating  the  number  of  additional  primary  aircraft 
flying  hours  that  would  be  required  before  the  fictional  pilot  would  be 
experienced  enough  to  fill  a  nonflying  staff  assignment  and  to  judge  the 
combat  readiness  of  each  fictional  pilot  by  estimating  the  number  of  spin- 
up  sorties  that  would  be  required  prior  to  being  deployed  to  combat. 

After  responding  to  all  the  pilot  profiles,  respondents  to  the  F-15 
survey  were  asked  to  estimate  numbers  and  types  of  training  sorties 
(including  simulation  sorties)  that  a  pilot  should  fly  each  month.  The 
experts  were  asked  to  give  their  estimate  of  the  optimal  and  minimal 
number  of  aircraft  sorties  (BFM,  ACM,  two-ship  ACT,  four-ship  ACT, 
Red  Flag,  LFE,  Red  Air,  and  AEF)  and  of  the  optimal  and  minimal 
number  of  high-fidelity  simulator  sorties  (ACM,  two-ship  ACT,  four- 
ship  ACT,  LFE)  that  should  be  flown  each  month. 


5  At  the  time  of  the  survey,  simulator  sorties  were  referred  to  as  distributed  mission  training 
(DMT)  sorties. 
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Findings  from  the  F-15  Survey 

For  the  question  of  readiness  for  a  staff  position,  a  multiple  linear  regres¬ 
sion  was  done  with  the  dependent  variable  being  the  number  of  addi¬ 
tional  hours  the  pilot  needed  to  be  ready  for  a  staff  position.  For  the 
combat  readiness  question,  a  multiple  linear  regression  was  done  with 
the  dependent  variable  being  the  number  of  spin-up  sorties  needed  to 
be  combat  ready. 

Readiness  for  a  Staff  Position 

Table  4.1  presents  our  survey  results  on  the  question  of  readiness  for  a 
staff  position.  These  results  show  that  expert  judgments  of  pilot  expe¬ 
rience,  in  terms  of  readiness  for  a  nonflying  staff  position,  depend  on 
more  than  just  flying  hours.  The  results  of  our  analysis  of  expert  judg¬ 
ments  of  a  pilot’s  readiness  for  a  staff  position  show  that  prior  experi¬ 
ence,  which  is  highly  correlated  with  the  number  of  flying  hours  a  pilot 
has,  is  one  of  the  main  determinants.  That  prior  experience,  not  just 
flying  hours,  influences  judgments  about  readiness  for  a  staff  position 
is  indicated  by  the  fact  that  the  coefficients  for  PMA  hours  and  total 
flying  hours  shown  in  Table  4.1  are  not  significantly  different  from 
zero,  while  the  coefficient  for  prior  experience  is  nonzero  and  signifi¬ 
cant  at  the  0.05  level. 

A  pilot’s  upgrade  level  also  had  a  significant  effect  on  judgments 
of  experience  in  terms  of  readiness  for  a  staff  position.  The  main  effect 
of  upgrade  level  came  from  the  distinction  between  wingman  and  all 
other  upgrade  levels.  This  result  is  shown  graphically  in  Figure  4.2, 
which  plots  the  average  judgments  about  readiness  for  a  staff  posi¬ 
tion  in  terms  of  the  additional  PMA  hours  required  as  a  function  of 
upgrade  level.  The  results  in  Figure  4.2  show  that,  on  average,  a  wing- 
man  needed  about  400  more  PMA  hours  than  a  flight  lead,  instructor 
pilot,  or  mission  commander  to  be  considered  ready  for  a  staff  posi¬ 
tion,  whereas  the  flight  lead,  instructor  pilot,  and  mission  commander 
needed  about  the  same  number  of  extra  PMA  hours  (400)  to  be  con¬ 
sidered  ready  for  a  staff  position.  These  results  suggest  that,  as  long  as 
a  pilot  remains  qualified  only  as  a  wingman,  he  or  she  would  never  be 


Experience  and  Development  Surveys  43 


Table  4.1 

F-15  Experience  Survey  Results:  Readiness  for  a  Staff  Position 


Regression  Statistics 


Multiple  R 

0.450980681 

Standard  error 

0.875334918 

R2 

0.203383574 

Adjusted  R2 

0.182099929 

Observations 

270 

Analysis  of  Variation 


df 

SS 

MS 

F  Significance  F 

Regression 

7 

51.25266071 

7.321808673 

9.555862007  1.42187  x  10“10 

Residual 

262 

200.7473393 

0.766211219 

Total 

269 

252 

95%  Confidence 


Coefficients 

Standard 

Error 

t- 

Statistic 

P- 

Value 

Lower 

Upper 

Intercept 

-2.78 

2.24 

-1.24 

0.22 

-7.20 

1.63 

PMA 

0.00 

0.00 

-0.66 

0.51 

0.00 

0.00 

Total 

0.00 

0.00 

-0.79 

0.43 

0.00 

0.00 

PMAx  upgrade  level 

0.00 

0.00 

1.02 

0.31 

0.00 

0.00 

Prior  experience 

0.41 

0.20 

2.08 

0.04 

0.02 

0.79 

Wingman 

4.86 

2.25 

2.17 

0.03 

0.44 

9.29 

Flight  lead 

3.34 

2.15 

1.55 

0.12 

-0.89 

7.58 

Instructor  pilot 

2.26 

1.71 

1.32 

0.19 

-1.11 

5.63 

NOTE:  The  dependent  variable  in  the  regression  is  the  estimate  of  the  number 
of  extra  hours  needed  to  be  ready  for  a  staff  position.  This  judgment  is  based  on 
information  about  the  pilot's  flying  hours  (PMA  and  total),  upgrade  level  (wingman 
or  flight  lead  or  instructor  pilot),  and  the  interaction  between  these  PMA  hours  and 
upgrade  level.  (The  analysis  did  not  include  the  interaction  between  total  hours 
and  upgrade  level  because  it  contributed  nothing  to  the  prediction  of  readiness  for 
a  staff  position.)  The  estimates  of  the  extra  hours  a  pilot  needed  to  be  considered 
experienced  were  standardized  within  subjects.  So,  the  regression  factors  out 
individual  differences  between  subjects  in  how  they  estimate  the  hours  needed  for 
a  staff  position. 
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Figure  4.2 

Additional  Primary  Aircraft  Hours  Required  to  Be  "Experienced" 
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considered  experienced,  in  the  sense  of  being  ready  for  a  staff  position, 
regardless  of  the  number  of  primary  aircraft  hours  flown.  The  results 
displayed  in  Figure  4.2  also  suggest  that  a  pilot  must  at  least  be  a  flight 
lead  to  be  considered  ready  for  a  staff  position. 

One  surprising  observation  based  on  the  results  in  Figure  4.2  is 
that  the  respondents  in  this  survey  indicated  that  a  pilot  should  have  far 
more  hours  than  RDTM  specifies  to  be  considered  sufficiently  expe¬ 
rienced  for  a  staff  position  (the  conventional  meaning  of  experienced 
from  the  RDTM  perspective).  The  average  number  of  PMA  hours  for 
the  fictional  flight  leads,  instructor  pilots,  and  mission  commanders  in 
this  survey  was  over  500,  so  the  number  of  additional  hours  the  experts 
recommended  for  these  pilots  to  be  considered  ready  for  staff  positions 
brought  the  average  total  hours  to  nearly  1,000,  which  is  twice  the 
RDTM  requirement  for  pilots  entering  units  after  their  initial  upgrade 
training  in  a  fighter  aircraft.  Clearly,  the  respondents  to  this  survey 
wanted  pilots,  like  themselves,  to  do  a  lot  of  flying  before  moving  to  a 
staff  job.  Flowever,  this  result  could  indicate  that  pilots  simply  think 
they  should  do  more  flying  at  the  time  in  their  careers  when  they  are  in 
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flying  units,  not  that  the  extra  flying  time  is  required  to  be  successful 
in  a  staff  position. 

The  results  displayed  in  Figure  4.2  are  also  interesting  because 
they  suggest  that  there  is  a  very  strong  feeling  among  pilots  that  the 
current  RDTM  system  is  sending  pilots  to  staff  positions  before  they 
are  experienced  enough.  What  the  experts  are  saying  in  this  survey 
seems  to  call  into  question  some  of  the  fundamental  assumptions  of 
the  RDTM  system.  However,  it  may  be  that  the  additional  number 
of  flying  hours  these  experts  judge  to  be  required  to  be  experienced 
beyond  the  current  RTDM  requirement  also  reflects  their  recent  expe¬ 
rience  with  the  quality  of  flying  hours — a  greater  proportion  of  flying 
time  spent  “boring  holes  in  the  sky.” 

Readiness  for  Combat 

Table  4.2  presents  our  survey  results  on  the  question  of  readiness  for 
combat.  We  looked  at  how  various  aspects  of  training  affected  judg¬ 
ments  of  pilot  experience  in  terms  of  combat  readiness.  The  results  of 
this  part  of  the  survey  showed  that  training  variables  had  a  significant 
influence  on  judgments  of  combat  readiness.  The  predictor  variables  in 
this  analysis  accounted  for  36  percent  of  the  variance  in  the  judgments 
(see  Table  4.2). 

Training  Variables.  For  this  analysis,  ten  measures  of  training  expe¬ 
rience  were  used  to  predict  judgments  about  pilot  experience  in  terms 
of  combat  readiness.  Judgments  about  combat  readiness  were  again 
made  in  terms  of  the  number  of  spin-up  sorties  required  before  deploy¬ 
ment  to  combat;  this  assumes  that  the  greater  the  number  of  spin-up 
sorties  required,  the  less  combat  ready  the  pilot.  The  training  variables 
used  in  the  analysis  included  high-fidelity  simulator  training.  The  pre¬ 
dictor  variables  were  measured  in  terms  of  the  number  of  the  different 
types  of  training  sorties  flown.  Our  analysis  of  judgments  about  combat 
readiness  included  the  following  types  of  aircraft  and  simulator  sorties: 
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Table  4.2 

F-15  Experience  Survey  Results:  Aircraft  Proficiency 


Regression  Statistics 

Multiple  R 

0.602 

Standard  error 

0.0.789 

R2 

0.362 

Adjusted  R2 

0.335 

Observations 

270 

Analysis  of  Variation 


df 

SS 

MS 

F  Significance  F 

Regression 

11 

91.20443544 

8.291312313 

13.30359194  4.8902  x  10“10 

Residual 

258 

160.7955646 

0.623238622 

Total 

269 

252 

95%  Confidence 


Coefficients 

Standard 

Error 

t- 

Statistic 

P- 

Value 

Lower 

Upper 

Intercept 

1.024 

0.542 

1.888 

0.060 

-0.044 

2.092 

Last  month 

Aircraft 

BFM 

-0.139 

0.087 

-1.599 

0.111 

-0.311 

0.032 

ACM 

-0.210 

0.066 

-3.199 

0.002 

-0.339 

-0.081 

ACT 

-0.261 

0.038 

-6.849 

0.000 

-0.336 

-0.186 

LFE 

-0.156 

0.026 

-5.889 

0.000 

-0.208 

-0.104 

Red  Air 

0.019 

0.054 

0.353 

0.724 

-0.087 

0.126 

Simulator 

ACT 

0.056 

0.049 

1.137 

0.257 

-0.041 

0.153 

LFE 

-0.126 

0.058 

-2.159 

0.032 

-0.240 

-0.011 

First  five  months 

Aircraft 

0.010 

0.010 

0.979 

0.328 

-0.010 

0.030 

AEF 

0.021 

0.010 

2.017 

0.045 

0.001 

0.042 

Simulator 

0.010 

0.015 

0.671 

0.503 

-0.020 

0.040 

NOTE:  The  dependent  variable  in  the  regression  is  the  estimate  of  the  number  of 
spin  up  sorties  required  for  combat  readiness.  The  predictors  are  the  number  of 
sorties  of  different  types  that  were  flown  in  the  last  month  prior  to  combat  or  in  the 
first  five  months  of  the  six  months  prior  to  combat. 
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•  Aircraft  sorties — the  number  of 

-  aircraft  BFM  sorties  flown  in  the  last  month  prior  to  deploy¬ 
ment 

-  aircraft  ACM  sorties  flown  in  the  last  month  prior  to  deploy¬ 
ment 

-  aircraft  ACT  sorties  (both  two-  and  four-ship)  flown  in  the  last 
month  prior  to  deployment 

-  aircraft  LFE  sorties  flown  in  the  last  month  prior  to  deploy¬ 
ment 

-  aircraft  Red  Air  sorties  flown  in  the  last  month  prior  to  deploy¬ 
ment 

-  all  aircraft  sorties,  of  all  types,  flown  in  the  five  months  prior  to 
the  last  month  before  deployment. 

—  AEF  sorties  flown  in  the  five  months  prior  to  the  last  month 
before  deployment 

•  Sorties  in  DMO-capable  simulator  systems — The  number  of 

-  high-fidelity  simulator  ACM  sorties  flown  in  the  last  month 
prior  to  deployment 

-  high-fidelity  simulator  ACT  sorties  flown  in  the  last  month 
prior  to  deployment 

-  high-fidelity  simulator  LFE  sorties  flown  in  the  last  month 
prior  to  deployment 

-  all  DMO-capable  simulator  sorties  of  all  types,  flown  in  the 
five  months  prior  to  the  last  month  before  deployment. 

Results.  Table  4.3  summarizes  the  results.  The  training  vari¬ 
ables  used  to  predict  judgments  of  combat  readiness  are  listed  in 
the  far  left  column.  The  significance  of  a  predictor  is  again  given  by 
the  P-value,  in  the  far  right  column.  Significant  predictors,  with  P- 
values  less  than  0.05,  are  shaded.  One  set  of  significant  predictors  of 
combat  readiness  consisted  of  the  numbers  of  aircraft  ACM,  ACT, 
and  LFE  sorties  flown  in  the  last  month  prior  to  combat  deploy¬ 
ment.  The  coefficients  for  all  these  predictors  are  negative,  indicating 
that  the  more  of  these  sorties  were  flown  in  the  last  month  prior  to 
deployment,  the  fewer  spin-up  sorties  were  required.  The  P-values  for 
these  three  predictors  show  that  all  are  significant  at  the  0.01  level. 
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Table  4.3 

Coefficients  of  Training  Variables  Used  in  Regression  Analysis 
with  Combat  Readiness  as  the  Dependent  Variable 


Type  of  Sortie 

Coefficients 

Standard 

Error 

t-Statistic 

P-Value 

Intecept 

1.024 

0.542 

1.888 

0.060 

Last  month 

Aircraft 

BFM 

-0.139 

0.087 

-1.599 

0.111 

ACM 

-0.210 

0.066 

-3.199 

0.002 

ACT 

-0.261 

0.038 

-6.849 

0.000 

LFE 

-0.156 

0.026 

-5.889 

0.000 

Red  Air 

0.019 

0.054 

0.353 

0.724 

Simulator 

ACT 

0.056 

0.049 

1.137 

0.257 

LFE 

-0.126 

0.058 

-2.159 

0.032 

First  five  months 

Aircraft 

0.010 

0.010 

0.979 

0.328 

AEF 

0.021 

0.010 

2.017 

0.045 

Simulator 

0.010 

0.015 

0.671 

0.503 

NOTE:  Significant  predictors,  with  P-values  less  than  0.05,  are  shaded. 


DMO-Capable  Simulator  Sorties  and  Combat  Readiness.  The 

only  other  significant  predictors  of  combat  readiness  in  Table  4.3  were 
the  number  of  AEF  sorties  flown  in  the  five  months  prior  to  the  last 
month  before  deployment  and  the  number  of  high-fidelity  simulator 
LFE  sorties  flown  in  the  last  month  prior  to  combat  deployment.  The 
positive  coefficient  for  AEF  sorties  shows  that  the  more  of  these  sor¬ 
ties  that  are  flown,  the  more  spin-up  sorties  are  needed  before  combat 
deployment.  This  result  is  consistent  with  the  opinion,  often  informally 
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expressed  by  pilots,  that  AEF  sorties  actually  provide  negative  training 
experience. 

Perhaps  the  most  interesting  result  of  this  survey  is  the  finding 
that  experts  see  recent  LFE  sorties  conducted  in  the  high-fidelity  simu¬ 
lator  as  contributing  significantly  to  the  development  of  pilot  experi¬ 
ence  measured  in  terms  of  combat  readiness.  The  negative  coefficient 
value  for  simulator  EFE  sorties  shows  that,  if  more  of  these  sorties  are 
flown  in  the  last  month  prior  to  deployment,  fewer  spin-up  sorties  are 
needed  before  deployment.  Thus,  without  being  asked  directly  about 
the  role  of  high-fidelity  simulators  in  fighter  training,  the  experts’  judg¬ 
ments  show  that  they  consider  high-fidelity  simulator  EFE  sorties,  if 
flown  in  the  month  immediately  prior  to  combat  deployment,  to  make 
a  significant  contribution  to  aircraft  proficiency,  measured  in  terms  of 
the  combat  readiness  of  fighter  pilots. 

The  results  of  the  survey  suggest  that  the  training  that  contributes 
most  significantly  to  aircraft  proficiency  is  that  received  in  the  month 
immediately  prior  to  deployment.  Both  aircraft  and  high-fidelity  simu¬ 
lator  training  contributes  to  aircraft  proficiency  if  it  is  received  during 
this  period.  However,  it  appears  that  the  only  kind  of  simulation  train¬ 
ing  that  was  judged  to  contribute  significantly  to  aircraft  proficiency, 
if  it  occurs  during  this  period,  is  LFE.  In  operational  training,  it  is 
impossible  to  know  in  advance  which  month  of  a  pilot’s  training  will 
be  the  last  month  before  combat  deployment;  the  survey  results  there¬ 
fore  suggest  that  it  would  be  worthwhile  to  have  a  pilot  fly  an  appropri¬ 
ate  number  of  simulated  LFE  sorties  every  month. 

Type  of  Training  and  Experience 

The  survey  results  show  that  ACM,  ACT,  and  LFE  (Red  Flag  and  other 
types  of  sorties)  are  the  main  contributors  to  the  development  of  combat 
readiness.  The  results  also  show  that  certain  types  of  sorties — AEF,  in 
particular — actually  reduce  combat  readiness.  This  result  shows  the 
potential  need  to  distinguish  between  two  types  of  experience:  experi¬ 
ence  that  prepares  one  for  a  staff  assignment  and  experience  that  prepares 
one  for  combat.  While  our  results  showed  that  AEF  sorties  are  negatively 
related  to  the  kind  of  experience  that  makes  a  pilot  combat  ready,  they 
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may  be  a  plus  for  getting  experience  suitable  for  staff  assignments,  since 
knowledge  of  deployed  operations  is  key  to  many  staff  jobs. 

The  results  of  the  F-15  survey  also  demonstrated  that  high- 
fidelity  simulator  training  itself,  particularly  training  involving  distrib¬ 
uted  multiship  simulation  exercises  (LFE),  contributes  to  the  develop¬ 
ment  of  pilot  experience.  This  is  shown  by  the  fact  that  the  number 
of  LFE  sorties  performed  in  simulators  during  continuation  training 
in  the  month  prior  to  deployment  was  judged  to  increase  the  pilot’s 
combat  readiness  significantly.  So,  a  mix  of  both  aircraft  and  high- 
fidelity  simulator  sorties  contributes  to  the  development  of  experience 
in  terms  of  aircraft  proficiency. 

We  can  get  an  idea  of  what  constitutes  an  appropriate  mix  of  air¬ 
craft  and  simulator  sorties  by  looking  at  experts’  ratings  of  the  optimal 
number  of  different  types  of  sorties  that  inexperienced  pilots  should  fly 
to  maintain  their  proficiency. 

Optimal  and  Minimal  Training  Sorties 

Figures  4.3  and  4.4  summarize  the  answers  to  the  survey  questions  on 
the  optimum  and  minimum  numbers  of  sorties  that  should  be  flown. 
Figure  4.3  shows  the  average  and  standard  deviation  of  the  ratings  of 
the  optimal  number  of  each  type  of  sortie  described  in  the  pilot  train¬ 
ing  histories:  aircraft  BFM,  aircraft  ACM,  aircraft  two-ship  ACT,  air¬ 
craft  four-ship  ACT,  aircraft  LFE  (Red  Flag  and  other  LFE),  aircraft 
Red  Air,  DMO  ACM,  DMO  two-ship  ACT,  DMO  four-ship  ACT, 
and  DMO  LFE.  The  experts  agreed  that  the  optimum  number  of  total 
aircraft  sorties  that  should  be  flown  each  month  was  between  12  and 
16,  with  the  average  being  14.  The  experts  were  also  quite  consistent 
(in  terms  of  the  small  standard  deviation  of  ratings  around  the  average) 
about  the  numbers  of  different  types  of  aircraft  sorties  that  should  be 
flown  each  month:  three  BFM,  two  ACM,  two  two-ship  ACT,  three 
four-ship  ACT,  two  LFE,  and  two  Red  Air. 

Respondents  also  agreed  that  the  optimum  number  of  high- 
fidelity  simulator  sorties  that  should  be  flown  each  month  was  between 
two  and  seven,  with  the  average  being  about  five.  The  respondents  also 
agree  that  high-fidelity  simulator  sorties  should  be  part  of  the  optimum 
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Figure  4.3 

The  Optimum  Numbers  of  Basic  Sortie  Types  That  Should  Be  Flown  Each 
Month  as  Part  of  Continuation  Training 


2vX  4vX  Air  A/C  ACM  ACT,  ACT,  DMO 

2vX  4vX  Sims 

Type 

RAND  MG597-4.3 

training  sortie  mix  and  were  quite  consistent  in  their  ratings  of  the 
optimum  numbers  of  different  types  of  simulator  sorties  that  should  be 
flown  each  month:  no  ACM,  two  two-ship  ACT,  two  four-ship  ACT, 
and  one  LFE. 

Figure  4.4  shows  that  the  experts  agreed  that  the  minimum 
number  of  aircraft  sorties  for  maintaining  proficiency  was  between 
seven  and  13,  with  the  average  near  ten,  four  less  than  the  optimum 
average.  The  main  cuts  in  sortie  types,  relative  to  the  optimum,  were 
in  aircraft  BFM  (cut  by  one  sortie),  aircraft  ACM  (cut  by  one  sortie), 
aircraft  four-ship  ACT  (cut  by  one  sortie),  and  aircraft  Red  Air  (cut  by 
one  sortie).  The  minimum  number  of  high-fidelity  simulator  sorties  is 
between  one  and  five,  with  an  average  of  about  three.  This  result  shows 
that  sorties  in  DMO-capable  simulator  systems  are  considered  an 
important  enough  part  of  training  to  be  included  even  in  the  minimum 
sortie  mix.  The  main  cuts  in  simulator  sorties  relative  to  the  optimum 
are  cuts  by  one  sortie  each  in  DMO  two-ship  ACT  and  four-ship  ACT. 
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Figure  4.4 

The  Minimum  Numbers  of  Basic  Training  Sortie  Types  That  Should  Be 
Flown  Each  Month  as  Part  of  Continuation  Training 


2vX  4vX  Air  A/C  ACM  ACT,  ACT,  DMO 


2vX  4vX  Sims 


Type 

RAND  MG597-4.4 


Insights  into  the  Meaning  of  Pilot  Experience 

To  be  considered  experienced  in  the  RDTM  sense  of  being  ready  for  a 
staff  assignment,  a  pilot  must  have  upgraded  at  least  to  flight  lead.  Until 
a  pilot  has  achieved  flight-lead  status,  increasing  flying  hours  produces 
very  little  increase  in  the  perceived  experience  level  of  the  pilot.  Of 
course,  this  is  consistent  with  the  notion  that,  if  a  pilot  does  not  achieve 
flight-lead  status  after  a  certain  amount  of  time,  the  pilot  probably  does 
not  have  “what  it  takes,”  no  matter  how  much  time  he  or  she  spends 
in  the  cockpit.  If  feasible,  increasing  the  speed  and  efficiency  of  pilot 
upgrades  would  increase  the  rate  at  which  pilots  could  become  experi¬ 
enced  in  terms  of  readiness  for  a  staff  position.  One  way  to  accelerate 
upgrades  could  be  to  use  simulation  to  better  prepare  pilots  for  upgrade 
sorties,  so  that  such  sorties  are  more  likely  to  be  completed  successfully. 
DMO -capable  simulator  systems  could  be  used  to  provide  the  kind 


Experience  and  Development  Surveys  53 


of  training  needed  to  prepare  pilots  for  the  multiship  upgrade  sorties 
needed  to  move  from  wingman  to  flight  lead  and  beyond. 

These  results  also  suggest  that  the  type  of  training  received — in 
terms  of  the  types  of  training  sorties  flown — matters  when  it  comes 
to  the  development  of  experience  in  the  sense  of  aircraft  proficiency. 
Advanced  aircraft  sorties  and  multiship  high-fidelity  simulator  sorties 
make  the  greatest  contribution  to  the  development  of  pilot  experience 
in  terms  of  aircraft  proficiency.  These  results  suggest  that  providing  an 
optimal  mix  of  appropriate  aircraft  and  high-fidelity  simulator  sorties 
could  develop  pilot  experience  in  terms  of  aircraft  proficiency.  Experts 
agree  that  about  26  percent  of  the  sorties  flown  in  this  optimal  mix  of 
sorties  would  be  conducted  in  the  high-fidelity  simulators. 


CHAPTER  FIVE 

Conclusions  and  Recommendations 


The  research  described  in  this  report  was  motivated  by  concerns  about 
the  Air  Force’s  ability  to  develop  experienced  pilots  at  a  rate  that  not 
only  maintains  an  appropriate  balance  of  experienced  and  inexpe¬ 
rienced  pilots  in  operational  fighter  units  but  also  prepares  pilots  to 
progress  and  develop  the  skills  required  to  fill  staff  and  supervisory 
positions.  The  problem  of  maintaining  this  balance  will  always  exist  in 
a  stressed  training  environment,  especially  when  experience  is  defined 
by  a  single  flying-hour  criterion  and  as  long  as  the  process  assumes 
that  continued  development  is  assured  once  the  500-hour  criterion  is 
met.  This  assumption  may  no  longer  be  valid  when  not  enough  flying 
hours  are  available  to  continue  the  development  of  experienced  pilots 
at  an  appropriate  rate.  Changes  over  time  in  the  quantity  and  quality 
of  training  available  in  the  operational  units  have  confirmed  that  the 
500-hour  criterion  has  not  had  consistent  meaning  for  describing  the 
training  and  combat  capabilities  pilots  possess  at  the  time  they  ini¬ 
tially  meet  that  criterion.  At  the  same  time,  high  pilot-production  goals 
have  generated  overmanning  and  low  experience  levels  in  many  fighter 
units.  As  the  Air  Force  is  seeking  ways  to  respond  to  these  problems,  it 
may  have  become  essential  to  decouple  pilot  absorption  from  contin¬ 
ued  pilot  development,  thereby  generating  a  requirement  to  monitor 
the  processes  independently.  Our  surveys,  conducted  between  Octo¬ 
ber  2002  and  October  2003,  helped  us  reexamine  the  concept  of  pilot 
experience  by  looking  at  what  factors  determine  how  operational  train¬ 
ing  experts  judge  what  constitutes  an  experienced  pilot  in  several  dis¬ 
tinct  contexts. 
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What  Is  Experience? 

The  results  of  this  research  suggest  that  flying  hours  give  only  a  rough 
approximation  of  what  experts  see  as  a  pilot’s  actual  experience  level. 
To  the  extent  that  experienced  means  readiness  for  supervisory  or  staff 
positions,  the  main  determinants  in  this  context  are  the  skills  and 
qualifications  required  for  the  increased  responsibilities  pilots  will  face 
in  these  positions.  These  factors  are  highly  correlated  with  flying  hours, 
but  our  surveys  showed  that  the  RDTM  definition  does  not  effectively 
measure  a  pilot’s  qualifications  for  staff  positions.  Our  experts  judge 
that,  for  pilots  to  be  considered  ready  for  a  staff  assignment,  they  need 
nearly  twice  as  many  flying  hours  as  would  make  them  experienced 
according  to  RDTM  standards.  This  casts  a  great  deal  of  doubt  on 
the  validity  of  the  previous  assumption  that  meeting  the  500-hour  cri¬ 
terion  automatically  enables  pilots  to  perform  effectively  or  continue 
with  their  development  in  nonflying  positions. 

Survey  results  also  show  that  the  type  of  training  received  makes 
a  big  difference  in  terms  of  the  development  of  the  experience  required 
to  develop  combat  capability  and  prepare  for  combat  operations.  Live 
training  sorties  are  the  strongest  contributors  to  reducing  the  spin-up 
training  required  to  prepare  for  imminent  combat  operations  in  a  spe¬ 
cific  theater,  so  flying  hours  will,  in  general,  be  highly  correlated  with 
the  development  of  this  essential  experience.  Not  all  flying  hours  are 
equal,  however,  in  terms  of  their  contribution  to  developing  the  expe¬ 
rience  that  will  be  required.  Indeed,  live  sorties  with  profiles  from  the 
high  end  of  the  training  hierarchy  contribute  more  to  the  preparation 
for  combat  than  sorties  from  the  low  end  of  the  hierarchy  and  much 
more  than  most  sorties  flown  in  support  of  a  different  AEF  tasking.  It 
should  be  noted,  however,  that  our  experts’  judgments  show  that  LFE 
sorties  conducted  in  a  high-fidelity  simulator  also  make  significant 
contributions  to  combat  preparation.  The  survey  results  also  suggest 
that  an  optimal  regime  for  the  development  and  maintenance  of  pilot 
experience  for  continuation  training  would  include  about  14  aircraft 
and  five  DMO  sorties  per  month.  Pilots  appear  to  endorse  consider¬ 
ing  certain  types  of  high-fidelity  simulator  time  when  determining  a 
person’s  qualification  for  certain  duties. 
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Developing  Qualified  Pilots 

The  survey  results  suggest  that  pilots  must  accumulate  a  great  deal  more 
experience  after  they  meet  the  initial  500-hour  criterion  if  they  are  to 
continue  to  develop  the  qualifications  required  for  meaningful  staff 
assignments.  This  strongly  suggests  that  the  Air  Force  needs  to  manage 
the  absorption  and  development  processes  separately  because  contin¬ 
ued  development  is  no  longer  ensured  following  the  initial  absorption 
period  during  their  first  operational  flying  opportunity. 

The  survey  results  also  indicate  that  live  training  that  is  supported 
and  augmented  with  appropriate  high-fidelity  simulator  training  pro¬ 
files  can  be  more  effective  in  preparing  pilots  for  imminent  combat 
operations  than  live  training  alone.  We  would  argue  that  the  advanced 
simulators  available  in  the  MTCs  need  to  be  formally  recognized  and 
required  in  the  applicable  training  directives.  This  would  help  ensure 
that  the  live  training  and  the  MTC  training  are  properly  coordinated 
to  improve  the  overall  training  efficiency  in  operational  units.  This 
process  can  benefit  substantially  from  the  existing  body  of  research  on 
advanced  simulator  use  that  has  been  conducted  at  AFRL  under  joint 
sponsorship  with  ACC. 

Unfortunately,  integrated  plans  leading  to  more  efficient  training 
are  only  feasible  in  weapon  systems  for  which  the  advanced  simulator 
and  MTC  facilities  are  universally  available.  This  will  require  resolv¬ 
ing  the  existing  funding  and  MTC  supervisory  issues  in  the  DMO- 
capable  facilities  and  keeping  the  operational  flight  program  software 
for  the  MTCs  consistent  with  that  of  the  aircraft. 

Nonetheless,  absorption  and  developmental  processes  are  still 
related  because  the  successful  completion  of  the  former  is  an  abso¬ 
lute  requirement  for  the  subsequent  success  of  the  latter.  Their  sep¬ 
arate  management,  however,  will  require  using  new  measures.  The 
absorption  process  can  perhaps  be  managed  by  modifying  the  exist¬ 
ing  RDTM  definition  to  incorporate  appropriate  MTC  training.1  The 
developmental  process,  however,  will  require  new  criteria  for  the  Air 


1  According  to  Air  Combat  Command  staff  members  (ACC/A3T),  this  is  already  being 
accomplished  for  aircraft  weapon  systems  with  appropriate  MTC  capability. 
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Force  to  measure  pilot  development  in  terms  of  the  additional  opera¬ 
tional  skills  and  qualifications  required  for  these  officers  to  perform 
staff  and  supervisory  duties.  The  Rated  Force  Development  Team  and 
its  associated  combat  air,  mobility  air,  and  special  operations  forces 
development  panels,  which  was  implemented  to  monitor  the  officer 
development  process,  could  provide  a  framework  to  oversee  this  course 
of  action,  but  rated  officers  may  need  to  be  reviewed  earlier  in  their 
careers  than  they  are  now  (which  is  as  they  reach  eligibility  for  promo¬ 
tion  to  major)  to  ensure  that  they  can  receive  the  required  second  and 
subsequent  flying  assignments  in  their  primary  aircraft. 


APPENDIX 


Survey  Results  from  F-16  Pilots  at  Hill  AFB 


Goals 

The  first  survey  was  specifically  designed  to  evaluate  how  various  train¬ 
ing  events  affected  the  development  of  experience  in  F-16  pilots.  This 
version  of  the  survey  did  not  include  information  about  high-fidelity 
simulator  systems  because  no  DMO  training  was  available  to  F-16 
pilots  at  the  time. 


Methods 

Each  expert  who  participated  in  the  survey  was  shown  a  set  of  20  fic¬ 
titious  but  realistic  pilot  profiles.1  Each  profile  contained  information 
about  the  pilot’s  overall  flight  training  history  and  training  sorties  flown 
during  the  six  months  prior  to  the  survey.  Information  about  the  pilot’s 
training  history  included  the  pilot’s  upgrade  level  (wingman,  flight  lead, 
or  instructor  pilot),  months  at  that  upgrade  level,  number  of  primary 
aircraft  hours,  and  total  number  of  flying  hours.  Information  about  the 
pilot’s  training  over  the  last  six  months  included  the  number  of  sorties 
flown  each  month;  the  average  duration  of  the  sorties  flown;  and  the 
total  number  of  air-to-air,  air-to-ground,  and  contingency  sorties  flown 
over  the  entire  six  months.  Figure  A.l  shows  an  example  of  the  screen 
display  for  one  pilot  that  a  participant  in  the  survey  would  see. 


1  During  their  service  at  RAND  as  Air  Force  Fellows,  knowledgeable  Air  Force  fighter 
pilots  reviewed  the  pilot  surveys  and  developed  data  for  the  profiles. 
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Figure  A.1 

Sample  Data  Screen  for  F-16  Survey 


RAND  MG597-A.1 


The  experts  rated  the  experience  level  of  the  pilots  described  in 
each  profile  in  two  ways:  first,  in  terms  of  readiness  for  a  staff  assign¬ 
ment  and,  second,  in  terms  of  readiness  for  combat.  We  did  not  ask 
directly  whether  a  pilot  described  in  a  profile  was  experienced.  We 
were  concerned  that  the  term  experienced  had  been  sufficiently  con¬ 
nected  to  specific  amounts  of  primary  aircraft  flying  hours  that,  if  we 
had  asked  whether  a  pilot  was  “experienced,”  the  expert  might  make 
the  judgment  based  solely  on  the  RDTM  criterion.  Instead,  we  asked 
the  experts  to  judge  whether  or  not  the  pilot  described  in  the  profile 
was  ready  to  be  assigned  to  a  nonflying  staff  position  (one  operational 
definition  of  experienced).  A  “yes”  response  (as  marked  in  the  upper 
right  of  Figure  A.  1)  was  taken  to  mean  that  the  expert  considered  the 
pilot  to  be  experienced.  Experts  also  rated  the  combat  readiness  of 
a  pilot  by  estimating  the  number  of  spin-up  sorties  the  pilot  needed 


Survey  Results  from  F-16  Pilots  at  Hill  AFB  61 


to  fly  before  being  deployed  to  combat  duty.  This  estimate  could  be 
made  in  the  bottom  section  of  the  survey  screen.  The  more  spin-up 
sorties  needed,  the  less  experienced  the  pilot  was  taken  to  be  in  terms 
of  combat  readiness. 

The  survey  was  sent  to  34  F-16  training  experts  at  Hill  AFB  and 
was  completed  and  returned  by  28,  giving  an  82-percent  return  rate. 
Training  experts  were  defined  as  squadron  commanders,  squadron  oper¬ 
ations  officers,  squadron  assistant  operations  officers,  squadron  weap¬ 
ons  officers,  squadron  training  officers,  wing  weapons  officers,  flight 
commanders,  and  instructor  pilots.  The  respondents  were  representa¬ 
tive  of  unit  training  experts  in  the  sense  that  the  number  of  respon¬ 
dents  from  these  different  positions  was  nearly  proportional  to  the  total 
number  of  people  in  those  positions.  The  28  experts  evaluated  a  total  of 
560  pilot  profiles — 20  profiles  per  expert.2 

Policy-capturing  was  used  to  determine  how  experts  used  infor¬ 
mation  about  a  pilot’s  overall  training  history  and  training  received 
over  the  last  six  months  to  form  judgments  about  the  pilot’s  experience 
in  terms  of  readiness  for  a  staff  position  and  combat  readiness. 

Separate  analyses  were  done  on  the  yes-or-no  judgments  of  a  pilot’s 
readiness  for  a  staff  position  and  the  quantitative  judgments  of  the 
pilot’s  combat  readiness,  but  in  both  cases,  the  statistical  analysis  was 
based  on  all  560  pilot  profile  evaluations.  For  the  question  of  readiness 
for  a  staff  position,  a  logistic  regression  with  PMA  hours,  non-PMA 
hours,  and  upgrade  level  (wingman,  flight  lead,  or  instructor  pilot)  as 
the  independent  variables  was  used  to  predict  the  yes-or-no  judgments 
of  the  experts.  For  the  judgment  of  combat  readiness,  multiple  regres¬ 
sion  analysis  was  used  with  the  number  of  spin-up  sorties  required  as 
the  dependent  variable  and  the  number  of  sorties  flown  each  month, 
the  average  sortie  duration  each  month,  and  the  number  of  air-to-air 
and  air-to-ground  sorties  as  the  independent  variables. 

Tables  A.l  and  A.  2  provide  the  detailed  statistical  results  for  the 
F-16  survey.  The  discussion  below  refers  to  the  results  presented  in 
these  two  tables. 


2  The  survey  program  generated  the  pilot  profiles  randomly,  so  each  expert  evaluated  20 
profiles  that  were  different  from  those  of  every  other  expert. 
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Table  A.1 

Detailed  Results  of  Initial  F-16  Survey  of  Experience  as  Readiness  for 
a  Staff  Position 


Odds 

Ratio 

Robust 

Standard 

Error 

95%  Confidence 

Experience 

Z 

P>  |z| 

Lower 

Upper 

Non-PMA  hours  (A) 

1.00 

0.001 

6.72 

0.00 

1.00 

1.01 

PMA  hours  (B) 

1.00 

0.000 

7.14 

0.00 

1.00 

1.00 

Flight  leader  or 
instructor  pilot  (C) 

2.41 

3.219 

0.66 

0.51 

0.17 

33.13 

A  x  C 

1.00 

0.001 

2.32 

0.02 

1.00 

1.00 

NOTES:  Logistic  regression  Number  of  observations  =  560 

WALD  chi2  (4)  =  106.68  Probability  >  chi2  =  0.0000 
LOG  pseudolikelihood  =  -101.0228  Pseudo  R2  =  0.7320 
Standard  error  adjusted  for  28  clusters  in  rater. 


Results 

Factors  that  Affect  Judgments  of  Readiness  for  Staff  Positions 

The  dependent  variable  in  the  regression  is  the  rating  of  whether  or  not 
a  pilot  is  ready  for  a  staff  assignment.  The  judgment  of  experience  is 
modeled  using  information  about  the  pilot’s  flying  hours  (PMA  hours 
and  non-PMA  hours),  upgrade  level  (flight  lead  or  instructor  pilot,  as 
opposed  to  wingman),  and  the  interaction  between  upgrade  level  and 
PMA  hours  in  the  data.  The  judgments  of  readiness  for  a  staff  position 
were  binary:  0  if  not  ready,  1  if  ready  for  a  staff  position.  As  a  result,  a 
logistic  regression  analysis  was  done.  Because  each  expert  judge  made 
multiple  nonindependent  judgments,  a  robust  standard  error  estimate 
was  used  (Huber,  1967). 

In  predicting  readiness  for  a  staff  position,  upgrade  level  was  ini¬ 
tially  coded  as  a  dummy  variable,  with  “instructor  pilot”  being  the  omit¬ 
ted  value.  However,  initial  results  indicated  that  upgrade  level  should 
be  collapsed  into  a  single  dummy  variable  of  “flight  lead  or  instructor 
pilot”  versus  “wingman.”  Table  A.l  shows  that  the  “odds  ratios”  of  PMA 
hours,  non-PMA  hours,  and  the  interaction  of  PMA  hours  and  upgrade 
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Table  A. 2 

Detailed  Results  of  Initial  F-16  Survey  of  Experience  as  Aircraft 
Proficiency 


Regression  Statistics 

Multiple  R 

0.38222543 

Standard  error 

0.20830882 

R2 

0.14609628 

Adjusted  R2 

0.13369841 

Observations 

560 

Analysis  of  Variance 


df 

SS 

MS 

F 

Significance  F 

Regression 

8 

4.09069589 

0.511336991 

1.7839766 

1.5015  x  10“10 

Residual 

551 

23.90930417 

0.04339257 

Total 

559 

28 

Sorties 

Standard 

Coefficients  Error  t-statistic  P-value 

95%  Confidence 

Lower  Upper 

Intercept 

-0.20 

0.09 

-2.17 

0.03 

-0.39 

-0.02 

During  month 

Two 

-0.16 

0.08 

-1.90 

0.06 

-0.32 

0.01 

Three 

-0.02 

0.05 

-0.41 

0.68 

-0.12 

0.08 

Four 

-0.04 

0.05 

-0.71 

0.48 

-0.13 

0.06 

Five 

0.11 

0.08 

1.35 

0.18 

-0.05 

0.27 

Six 

0.00 

0.03 

0.01 

1.00 

-0.05 

0.05 

Air-to-air 

0.02 

0.02 

1.12 

0.26 

-0.02 

0.07 

Air-to-ground 

0.02 

0.02 

0.96 

0.34 

-0.02 

0.06 

level  are  all  statistically  significant;  unfortunately,  the  values  are  1, 
which  means  that  they  are  statistically  independent  of  the  “ready 
for  staff  duty”  dependent  variable.  The  box-and-whisker  diagram  in 
Figure  A.2  does  hint,  however,  that  experts  base  their  judgments  about 
readiness  for  a  staff  position  on  both  flying  hours  and  upgrade  level. 
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Figure  A. 2 

PMA  Hours  by  Upgrade  Status  and  Judged  Experience 
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Figure  A.2  provides  median,  maximum,  minimum,  and  interquartile 
range  for  the  number  of  PMA  hours  needed  to  be  judged  by  the  survey 
respondent  as  experienced  enough  for  a  staff  position.  The  median 
number  of  PMA  hours  for  a  wingman  to  be  judged  experienced  was 
over  2,000;  the  median  number  of  PMA  hours  for  a  flight  lead  or 
instructor  pilot  to  be  considered  experienced  was  about  1,700. 

Factors  That  Affect  Judgments  of  Combat  Readiness 

The  dependent  variable  in  the  regression  is  the  estimate  of  the  number 
of  spin-up  sorties  required  for  combat  readiness.  The  predictors  are  the 
number  of  sorties  flown  in  each  month  in  the  six  months  prior  to  the 
present,  the  number  of  air-to-air  (and  air-to-ground)  sorties  flown,  and 
the  experience  level  of  the  pilot. 

The  factors  included  in  this  analysis  were  all  measures  of  training 
the  pilots  had  received  over  the  last  six  months:  the  total  number  of  sor¬ 
ties  the  pilot  had  flown  each  of  the  immediately  preceding  six  months 
and  the  total  number  of  air-to-air  and  air-to-ground  sorties  flown  in 
the  six-month  period  prior  to  the  present.  The  P-values  in  Table  A.2 
show  that  none  of  these  training  factors  was  a  significant  predictor  of 
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combat  readiness.  Moreover,  the  R2  value  of  0.14  (shown  as  one  of  the 
regression  statistics)  indicates  that  the  training  factors  included  in  this 
analysis  accounted  for  only  a  small  proportion  of  the  variation  in  judg¬ 
ments  of  combat  readiness. 

A  closer  look  at  the  data  provided  some  insight  into  why  none  of  the 
training  factors  was  significant.  Analysis  of  the  judgments  of  individual 
experts  revealed  large  individual  differences  across  experts  in  terms  of 
how  information  about  a  pilot’s  recent  training  history  was  used  to  judge 
combat  readiness.  These  individual  differences  washed  out  any  effects 
of  training  variables  when  the  judgments  of  all  experts  were  included  in 
the  policy-capturing  analysis.  We  suspected  that  the  differences  among 
experts  in  terms  of  how  they  judged  combat  readiness  might  have  resulted 
from  their  need  to  imagine  facts  about  the  fictional  pilots  described  in 
the  profiles  that  were  not  provided  in  the  survey,  with  different  experts 
imagining  different  facts.  Therefore,  we  designed  the  next,  and  final,  ver¬ 
sion  of  the  survey  to  give  more  detail  about  each  pilot  and  each  pilot’s 
training  history.  Also,  because  the  final  survey  was  to  be  given  to  F-15 
training  experts  who  had  experience  with  DMO-capable  systems  as  part 
of  their  training,  the  final  survey  also  included  information  about  the 
number  of  predeployment  simulation  sorties  flown  by  each  pilot  in  the 
MTC  where  such  training  occurs. 
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